Elamur Research⌕
☰
home / papers / 2026 / Multi-dimensional Evaluation of Character-Authentic Dialogue Models Learned from Question-Answer Data

Multi-dimensional Evaluation of Character-Authentic Dialogue Models Learned from Question-Answer Data

Summary

LLMs that role-play a specific character tend to slip into a generic voice and fabricate persona facts, mostly because character-specific training data is scarce. Otsuka et al. (NTT) use question–answer (QA) collections as the training resource, since surveys or role-playing crowdworkers can supply them cheaply. An LLM turns the QAs into a short character profile (CP) and a pool of inferred character knowledge (CK) sentences, which are retrieved via RAG. The study compares Wikipedia-based baselines, zero/few-shot prompting, SFT, DPO, and a hybrid of CP/CK retrieval with DPO, in both single- and multi-turn settings. Evaluation covers reproducibility, diversity, hallucination, and character authenticity. Authenticity is scored by an LLM judge using trait rubrics derived from the data. SFT matches references best and hallucinates least, but its replies are short and plain. DPO sounds more in character but needs external knowledge to stay factual. The retrieval-plus-DPO hybrid gives the best overall balance and offers practical guidance for building character agents from scalable QA data.

Key points

  • Compared systems: [WIKI] (Wikipedia-description prompt, following Wang et al., 2024b), [WIKI+QA] (adds similar QAs retrieved by RAG), CP/CK prompting, [SFT], DPO, and the hybrid CP+CK+DPO model. Fine-tuning uses LoRA.
  • CK is built by prompting an LLM to infer explicit facts, implied relations, and causal links from each QA answer. New knowledge can be added to the store after training without retraining the model.
  • DPO training data pairs the character's real QA answer with responses generated by the base LLM and by the SFT model. The real answer is the preferred output.
  • Character authenticity is judged by an LLM-as-a-Judge in two steps. First, a definition of the character's traits is produced from a description and the QAs. Then a 10-point rubric is applied, with the judge acting as a fan of the character.
  • The authors also analyze how sensitive the models are to knowledge retrieval and response length. A noted risk is that the terse QA style can push models toward overly short multi-turn replies. Published at LREC 2026, pp. 2667–2681.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4ixiaqyfsd93

@inproceedings{otsuka2026multi,
  title = {{Multi-dimensional Evaluation of Character-Authentic Dialogue Models Learned from Question-Answer Data}},
  author = {Otsuka, Atsushi and Matsuo, Kazuya and Hama, Kenta and Mizukami, Masahiro and Arimoto, Tsunehiro and Sugiyama, Hiroaki and Nakatsuji, Makoto and Nomoto, Narichika},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4ixiaqyfsd93},
  url = {https://doi.org/10.63317/4ixiaqyfsd93}
}

References

33 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jaewoo Ahn, Taehyun Lee, Junyoung Lim, Jin-Hwa Kim, Sangdoo Yun, Hwaran Lee, and Gunhee Kim. 2024. TimeChara: Evaluating point-in-time character hallucination of role-playing large language models. In Findings of the Association for Computational Linguistics (ACL), pages 3291– 3325, Bangkok, Thailan… find
  2. Simone Alghisi, Massimo Rizzoli, Gabriel Roccabruna, Seyed Mahed Mousavi, and Giuseppe Riccardi. 2024. Should we fine-tune or RAG? evaluating different techniques to adapt LLMs for dialogue. In Proceedings of the 17th International Natural Language Generation Conference (INLG), pages 180–197. find
  3. Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data creation with 1,000,000,000 personas. find our card
  4. Character.AI. 2022. Introducing character. Blog post. find
  5. CohereForAI. 2024. Model card for c4ai command r+. find
  6. Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022. On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… find our card
  7. Ryuichiro Higashinaka, Masahiro Mizukami, Hidetoshi Kawabata, Emi Yamaguchi, Noritake Adachi, and Junji Tomita. 2018. Role play-based question-answering by real users for building chatbots with consistent personalities. In Proceedings of the 19th Annual Meeting of the Special Interest Group on Disco… find
  8. Masato Hirakawa, Shintaro Horie, Tomoaki Nakamura, Daisuke Oba, Sam Passaglia, and Akira Sasaki. 2024. elyza/llama-3-elyza-jp-8b. find
  9. Pegah Jandaghi, Xianghai Sheng, Xinyi Bai, Jay Pujara, and Hakim Sidahmed. 2024. Faithful persona-based conversational dataset generation with large language models. In Proceedings of the 6th Workshop on NLP for Conversational AI (NLP4ConvAI), pages 114–139. Association for Computational Linguistics… find
  10. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12). find our card
  11. Che Jiang, Biqing Qi, Xiangyu Hong, Dayuan Fu, Yang Cheng, Fandong Meng, Mo Yu, Bowen Zhou, and Jie Zhou. 2024. On large language models’ hallucination with regard to known facts. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hu… find
  12. Cheongwoong Kang and Jaesik Choi. 2023. Impact of co-occurrence on factual knowledge of large language models. In Findings of the Association for Computational Linguistics (EMNLP), pages 7721–7735. find
  13. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 110–119. find our card
  14. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81. Association for Computational Linguistics. find
  15. Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024. Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 7828– 7840. Association for C… find
  16. Koh Mitsuda, Ryuichiro Higashinaka, Hiroaki Sugiyama, Masahiro Mizukami, Tetsuya Kineb-uchi, Ryuta Nakamura, Noritake Adachi, and Hidetoshi Kawabata. 2022. Fine-tuning a pre-trained transformer-based encoder-decoder model with user-generated question-answer pairs to realize character-like chatbots.… find
  17. OpenAI. 2023. Chatgpt: Chat generative pre-trained transformer. https://openai.com/chatgpt. ↗
  18. Atsushi Otsuka, Kazuya Matsuo, Ryo Ishii, Narichika Nomoto, and Hiroaki Sugiyama. 2024. User-specific dialogue generation with user profile-aware pre-training model and parameter-efficient fine-tuning. Computing Research Repository, arXiv:2409.00887. — [arXiv:2409.00887](https://arxiv.org/abs/2409.00887) find
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computa… find our card
  20. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728– 53741. Curran Associates, Inc. find
  21. Stephen Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at trec-3. pages 0–. find our card
  22. Amir Saeidi, Shivanshu Verma, Md Nayem Uddin, and Chitta Baral. 2025. Insights into alignment: Evaluating DPO and its variants across multiple tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL, Volume 4: Student Research Workshop), pages 409–421. find
  23. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-LLM: A trainable agent for role-playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13153–13187, Singapore. Association for Computational Linguistics (ACL). find our card
  24. Hiroaki Sugiyama, Toyomi Meguro, Ryuichiro Higashinaka, and Yasuhiro Minami. 2014. Large-scale collection and analysis of personal question-answer pairs for conversational agents. In Intelligent Virtual Agents, pages 420–433. Springer International Publishing. find
  25. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. find our card
  26. Xi Wang, Hongliang Dai, Shen Gao, and Piji Li. 2024b. Characteristic AI agents via large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), pages 3016–3027. find
  27. Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, and Yanghua Xiao. 2024c. InCharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Proceedings of the 62nd Annual… find
  28. Chen Xu, Piji Li, Wei Wang, Haoran Yang, Siyun Wang, and Chuangbai Xiao. 2022. Cosplay: Concept set guided personalized dialogue generation across both party personas. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), SIGIR ’2… find
  29. Yeyong Yu, Runsheng Yu, Haojie Wei, Zhanqiu Zhang, and Quan Qian. 2025. Beyond dialogue: A profile-dialogue alignment framework towards general role-playing language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11992… find
  30. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2204– 2213. find our card
  31. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In Proceedings of the 8th International Conference on Learning Representations (ICLR). find
  32. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neu… find
  33. Junkai Zhou, Liang Pang, Huawei Shen, and Xueqi Cheng. 2023. SimOAP: Improve coherence and consistency in persona-based dialogue generation via over-sampling and post-evaluation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pages 9945–9959, Toront… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/otsuka-2026-multi-dimensional-evaluation-character-authentic/