Elamur Research⌕
☰
home / papers / 2026 / Rethinking Role-Playing Evaluation: Anonymous Benchmarking and A Systematic Study of Personality Effects

Rethinking Role-Playing Evaluation: Anonymous Benchmarking and A Systematic Study of Personality Effects

Summary

Role-playing agents (RPAs) built on LLMs are usually benchmarked on famous fictional characters. Because these characters are well covered in pretraining data, high scores may reflect memorized knowledge rather than genuine role-playing skill. Peng and Chen propose an anonymized evaluation protocol to separate the two. The simulated character's name is replaced with a placeholder token in the prompt, and the original name is restored before scoring. Models must therefore work only from the supplied profile and dialogue.

Tests on CharacterEval and RoleAgentBench with three LLMs show that hiding names generally lowers performance, which suggests names act as hidden cues. To recover role fidelity, the authors add personality descriptions (MBTI and Big Five) to the prompt. These come either from human annotations in The Personality Database (PDB) or from the model itself, via self-report or interview-style generation. Personality augmentation improves results consistently across benchmarks, languages and models. Model-generated personalities perform about as well as human-annotated ones. This offers a fairer benchmark and a cheap way to build RPAs for lesser-known or real people.

Key points

  • Anonymization method: the target character's name is replaced with <anonymous character>. Other characters, such as Ron and Hermione when Harry is simulated, keep their names.
  • Benchmarks:
    • CharacterEval: Chinese, 8,032 questions, 77 characters (43 on PDB), 11 objectives scored 1–5 by a reward model.
    • RoleAgentBench: English and Chinese, General Response (1,005) and Summarization (462) tasks, 54 characters (39 on PDB), scored with LLM-as-a-judge win rates.
  • Effect of anonymization: models tested were gemini-2.0-flash, llama-3.1-405B-instruct and gpt-4o. For gpt-4o, the drops were statistically significant on Knowledge-Hallucination (2.839 → 2.607), Persona-Behavior and Persona-Utterance. Llama's average was essentially unchanged.
  • Human evaluation: two psychology graduate students compared outputs pairwise for 5 Harry Potter characters. Cohen's kappa was 0.415 (summary) and 0.308 (general response). They confirmed that personality-augmented RPAs beat the original condition.
  • Limitations and code: identity can still leak through nicknames, interlocutors or iconic traits, and the size of the gains differs by model and personality framework. Code is available at github.com/MiuLab/RolePlayBench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigdial-1.15

@inproceedings{peng2026rethinking,
  title = {{Rethinking Role-Playing Evaluation: Anonymous Benchmarking and A Systematic Study of Personality Effects}},
  author = {Peng, Ji-Lun and Chen, Yun-Nung},
  booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2026},
  doi = {10.18653/v1/2026.sigdial-1.15},
  url = {https://doi.org/10.18653/v1/2026.sigdial-1.15}
}

References

37 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chaoran Chen, Bingsheng Yao, Ruishi Zou, Wenyue Hua, Weimin Lyu, Yanfang Ye, Toby Jia-Jun Li, and Dakuo Wang. 2025. Towards a design guideline for rpa evaluation: A survey of large language model-based role-playing agents. Preprint, arXiv:2502.13012. find
  2. Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. 2024. From persona to personalization: A survey on role-playing language agents. Preprint, a… find our card
  3. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2025. Length-controlled alpacaeval: A simple way to debias automatic evaluators. Preprint, arXiv:2404.04475. find
  4. Zhenpeng Gao, Xiaofen Xin, and Xiangmin Xu. 2025. Tailorrpa: A retrieval-based framework for eliciting personalized and coherent role-playing agents in general domain. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 5381–5412. find
  5. Kai He, Yucheng Huang, Wenqing Wang, Delong Ran, Dongming Sheng, Junxuan Huang, Qika Lin, Jiaxing Xu, Wenqiang Liu, and Mengling Feng. 2025. Crab: A novel configurable role-playing llm with assessing benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistic… find
  6. Yuxuan Huang. 2024. Orca: Enhancing role-playing abilities of large language models by integrating personality traits. arXiv preprint arXiv:2411.10006. — [arXiv:2411.10006](https://arxiv.org/abs/2411.10006) find
  7. Oliver P John, Eileen M Donahue, and Robert L Kentle. 1991. Big five inventory. Journal of personality and social psychology. find
  8. Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, and 1 others. 2023. Chatharuhi: Reviving anime character in reality via large language model. arXiv preprint arXiv:2308.09597. find
  9. Cheng Liu, Yifei Lu, Fanghua Ye, Jian Li, Xingyu Chen, Feiliang Ren, Zhaopeng Tu, and Xiaolong Li. 2025. Cogdual: Enhancing dual cognition of llms via reinforcement learning with implicit rule-based rewards. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p… find
  10. Jiaheng Liu, Zehao Ni, Haoran Que, Tao Sun, Noah Wang, Jian Yang, JiakaiWang, Hongcheng Guo, Z.Y. Peng, Ge Zhang, Jiayi Tian, Xingyuan Bu, Ke Xu, Wenge Rong, Junran Peng, and Zhaoxiang Zhang. 2024. RoleAgent: Building, interacting, and benchmarking high-quality role-playing agents from scripts. In T… find
  11. Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024. Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment. arXiv preprint arXiv:2401.12474. find
  12. Yanqi Dai, Huanran Hu, Lei Wang, Shengjie Jin, Xu Chen, and Zhiwu Lu. 2024. Mmrole: A comprehensive framework for developing and evaluating multimodal role-playing agents. arXiv preprint arXiv:2408.04203. — [arXiv:2408.04203](https://arxiv.org/abs/2408.04203) find
  13. Robert R McCrae and Paul T Costa Jr. 1997. Personality trait structure as a human universal. American psychologist, 52(5):509. find
  14. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282. find
  15. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22. find
  16. Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S Bernstein. 2024. Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10109. find
  17. Ji-Lun Peng, Sijia Cheng, Egil Diau, Yung-Yu Shih, Po-Heng Chen, Yen-Ting Lin, and Yun-Nung Chen. 2024. A survey of useful llm evaluation. arXiv preprint arXiv:2406.00936. — [arXiv:2406.00936](https://arxiv.org/abs/2406.00936) find
  18. David J Pittenger. 1993. The utility of the myers-briggs type indicator. Review of educational research, 63(4):467–488. find
  19. Haiming Qin, Jiwei Zhang, Wei Zhang, KeZhong Lu, Mingyang Zhou, Hao Liao, and Rui Mao. 2025. R-char: A metacognition-driven framework for role-playing in large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26984–27002. find
  20. Gregory Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality traits in large language models. find our card
  21. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-llm: A trainable agent for role-playing. Preprint, arXiv:2310.10158. find our card
  22. Quan Shi, Carlos E Jimenez, Stephen Dong, Brian Seo, Caden Yao, Adam Kelch, and Karthik Narasimhan. 2025. Impersona: Evaluating individual level lm impersonation. arXiv preprint arXiv:2504.04332. — [arXiv:2504.04332](https://arxiv.org/abs/2504.04332) find
  23. Mirac Suzgun and Adam Tauman Kalai. 2024. Meta-prompting: Enhancing language models with task-agnostic scaffolding. arXiv preprint arXiv:2401.12954. find
  24. Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei-Lin Chen, Chao-Wei Huang, Yu Meng, and Yun-Nung Chen. 2024. Two tales of persona in llms: A survey of role-playing and personalization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16612–16631. 62nd Annual Meeting of… find our card
  25. Lei Wang, Jianxun Lian, Yi Huang, Yanqi Dai, Haoxuan Li, Xu Chen, Xing Xie, and Ji-Rong Wen. 2025a. Characterbox: Evaluating the role-playing capabilities of llms in text-based virtual worlds. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computa… find
  26. Noah Wang, Zy Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, and 1 others. 2024a. RoleLLM: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Findings of the Association for Computational Linguistics… find our card
  27. Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen-tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Shuchang Zhou, and 1 others. 2025b. Coser: Coordinating llm-based persona simulation of established roles. In Forty-second International Conference on Machine Learning. find
  28. Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, and 1 others. 2024b. InCharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Proceedings of the 62nd Annual Meeting of the Association fo… find
  29. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300. find our card
  30. Bowen Wu, Kaili Sun, Ziwei Bai, Ying Li, and Baoxun Wang. 2025. Raiden benchmark: Evaluating role-playing conversational agents with measurement-driven custom dialogues. In Proceedings of the 31st International Conference on Computational Linguistics, pages 11086–11106. find
  31. Rui Xu, Dakuan Lu, Xiaoyu Tan, Xintao Wang, Siyu Yuan, Jiangjie Chen, Wei Chu, and Yinghui Xu. 2024. Mindecho: Role-playing language agents for key opinion leaders. arXiv preprint arXiv:2407.05305. — [arXiv:2407.05305](https://arxiv.org/abs/2407.05305) find
  32. Quan Tu, Chuanqi Chen, Jinpeng Li, Yanran Li, Shuo Shang, Dongyan Zhao, Ran Wang, and Rui Yan. 2023. Characterchat: Learning towards conversational ai with personalized social support. arXiv preprint arXiv:2308.10278. — [arXiv:2308.10278](https://arxiv.org/abs/2308.10278) find
  33. Shihao Yang, Zhicong Lu, Yong Yang, Bo Lv, Yang Shen, and Nayu Liu. 2025. Hycora: Hyper-contrastive role-adaptive learning for role-playing. arXiv preprint arXiv:2511.08017. — [arXiv:2511.08017](https://arxiv.org/abs/2511.08017) find
  34. Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, and Rui Yan. 2024. CharacterEval: A chinese benchmark for role-playing conversational agent evaluation. In Proceedings of the Xinge Ye, Rui Wang, Yuchuan Wu, Victor Ma, Feiteng Fang, Fei Huang, and Yongbin Li. 2025. Cpo: Addressin… find
  35. Yeyong Yu, Runsheng Yu, Haojie Wei, Zhanqiu Zhang, and Quan Qian. 2024. Beyond dialogue: A profile-dialogue alignment framework towards general role-playing language model. Preprint, arXiv:2408.10903. find
  36. Yeyong Yu, Runsheng Yu, Haojie Wei, Zhanqiu Zhang, and Quan Qian. 2025. Beyond dialogue: A profile-dialogue alignment framework towards general role-playing language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11992… find
  37. Haonan Zhang, Run Luo, Xiong Liu, Yuchuan Wu, Ting-En Lin, Pengpeng Zeng, Qiang Qu, Feiteng Fang, Min Yang, Lianli Gao, and 1 others. 2025a. Omnicharacter: Towards immersive role-playing agents with seamless speech-language personality interaction. In Proceedings of the 63rd Annual Meeting of the As… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2026-rethinking-role-playing-evaluation-anonymous/