推理型大语言模型角色扮演能力评估与优化

Summary

The paper asks whether stronger reasoning helps large language models play characters. The authors ran a large empirical study with 26 models: 3 closed API models and 23 open models from six developers, including GPT-5, GPT-4 Turbo, Qwen2.5, Llama, gemma-2, Mistral and DeepSeek-R1 variants. They used six role-playing benchmarks in Chinese and English. Three settings were compared: plain zero-shot role-playing, role-playing with Chain-of-Thought prompting, and role-playing with dedicated reasoning models (LRMs). Scoring combined automatic metrics with LLM-based judging.

Contrary to common expectations, reasoning often lowered role-playing quality and broke the usual pattern of larger models scoring higher. A manual error analysis linked the losses to two causes: the model's focus moving from staying in character to getting the task right, and its wording becoming formal and generic. Models also remain weak on advanced character tasks.

The authors propose role-aware CoT and role-playing-oriented reinforcement learning. They refine the latter into the FCR-RL framework and report experiments supporting both directions. All benchmarks are packaged into OpenCompass so the evaluation can be reproduced in one step.

Key points

• Benchmarks: RoleBench (RoleLLM), HPD, CharacterEval, CroSS-MR, SocialBench and InCharacter (16Personalities and BFI questionnaires reported).
• Metrics: ROUGE-L, accuracy and exact match, plus a pre-trained reward model with 12 dimensions for CharacterEval and prompt-based LLM-as-a-Judge for InCharacter.
• Where CoT helped or hurt: Across 19 models, CoT tended to reduce scores on CroSS-MR, HPD, SocialBench and CharacterEval. It improved scores on InCharacter and RoleBench.
• Failure analysis: In failure cases sampled from Qwen2.5-7B-Instruct, over 70% of performance drops were attributed to "attention diversion", "linguistic style drift", or both. A more elaborate "Advanced CoT" prompt performed about the same as the standard CoT prompt on CroSS-MR.
• Other findings:
  • Reasoning-specialised models underperform at role-playing.
  • The Qwen series gave the best overall results.
  • Chinese role-playing was generally stronger than English.
  • Code is released at xcfcode/rolecompass.