Elamur Research⌕
☰
home / papers / 2023 / CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Models

CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Models

Paper · 2023 · Jinfeng Zhou, Zhuang Chen, Dazhen Wan et al.

Summary

General-purpose LLMs are mostly trained on short, task-oriented exchanges. As a result, they handle long social conversations poorly, especially when asked to stay in a specific persona. Prompting them to role-play tends to lose consistency and liveliness over many turns. Classic persona-based dialogue covers only shallow facts such as names or hobbies. The authors define a broader task, Character-based Dialogue (CharacterDial). In it, a character is described by attributes (identity, interests, viewpoints, experiences, social relationships) and by behaviors (linguistic style, emotional expression, interaction patterns). They crowdsourced a large Chinese CharacterDial corpus covering many character types and topics, pairing every session with a character profile. On this data they adapted ChatGLM with dedicated training and self-refinement steps, producing the CharacterGLM family at 6B–66B parameters. Human judges rated the models on consistency, human-likeness and engagement. By these ratings, CharacterGLM outperforms most closed-source LLMs, including the GPT series, and its advantage grows in later turns. Since the leading commercial system, Character.AI, is proprietary, an open model and dataset fill a real gap for research on customizable social agents.

Key points

  • CharacterGLM is built on ChatGLM in sizes from 6B to 66B. CharacterGLM-6B is openly released (GitHub thu-coai/CharacterGLM-6B; Hugging Face LingxinAI/CharacterGLM-6b), and the larger versions are available only via API.
  • The released subset of the Chinese CharacterDial corpus contains 1,034 high-quality dialogue sessions covering 250 characters, ranging from fictional (Sun Wukong) to real (Lu Xun).
  • The comparison with GPT-3.5 involved 10 annotators, each creating 4 characters. They held conversations of at least 20 turns with both models and judged every turn on consistency, human-likeness and engagement.
  • Against MiniMax, CharacterGLM-66B's overall net advantage rises with dialogue length, from −3 in turns 1–5 to +7 in turns 16–20. Judges generally favoured longer responses, which partly explains MiniMax's strength in early turns.
  • The authors list four open challenges: long-term memory and character growth, self-awareness of knowledge limits, character-to-character "societies," and modeling deeper cognition such as theory of mind.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🎓Renmin University of ChinaBeijing · China35 works · 14,757 citations🏢Zhipu AIBeijing · China12 works · 2,069 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Dialogue systems · 401 works

Cite

Cite as Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Jie Tang and Minlie Huang. 2023. CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Models. arXiv:2311.16832.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.16832 · arXiv 2311.16832

@misc{zhou2023characterglm,
  title = {{CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Models}},
  author = {Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Jie Tang and Minlie Huang},
  year = {2023},
  doi = {10.48550/arxiv.2311.16832},
  eprint = {2311.16832},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.16832}
}

References

44 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs/2001.09977, 2020. URL https://arxiv.org/abs/2001. 09977. ↗ our card
  2. Anthropic. Introducing claude, 2023. introducing-claude. find
  3. URL https://www.anthropic.com/index/John Langshaw Austin. How to do things with words, volume 88. Oxford university press, 1975. ↗
  4. Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, and Xinchao Xu. PLATO-2: towards building an open-domain chatbot via curriculum learning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Findings of the Association for Computational Linguisti… our card
  5. Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhihua Wu, Zhen Guo, Hua Lu, Xinxian Huang, Xin Tian, Xinchao Xu, Yingzhan Lin, and Zheng-Yu Niu. PLATO-XL: exploring the large-scale pre-training of dialogue generation. In Yulan He, Heng Ji, Yang Liu, Sujian Li, Chia-Hui Chang, Soujan… find
  6. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions. 2023. find
  7. Timothy W Bickmore and Rosalind W Picard. Establishing and maintaining long-term human-computer relationships. ACM Transactions on Computer-Human Interaction (TOCHI), 12(2): 293–327, 2005. find
  8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… our card
  9. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335, 2022… our card
  10. Terrence Fong, Illah Nourbakhsh, and Kerstin Dautenhahn. A survey of socially interactive robots. Robotics and autonomous systems, 42(3-4):143–166, 2003. find
  11. Chris Frith and Uta Frith. Theory of mind. Current biology, 15(17):R644–R645, 2005. find
  12. Google. An overview of bard: an early experiment with generative ai, 2023. find
  13. Herbert P Grice. Logic and conversation. In Speech acts, pages 41–58. Brill, 1975. find
  14. Ted Grover, Kael Rowan, Jina Suh, Daniel McDuff, and Mary Czerwinski. Design and evaluation of intelligent agent prototypes for assistance with focus and productivity at work. In Proceedings of the 25th international conference on intelligent user interfaces, pages 390–400, 2020. find
  15. Yuxian Gu, Jiaxin Wen, Hao Sun, Yi Song, Pei Ke, Chujie Zheng, Zheng Zhang, Jianzhu Yao, Lei Liu, Xiaoyan Zhu, and Minlie Huang. EVA2.0: investigating open-domain chinese dialogue systems with large-scale pre-training. Mach. Intell. Res., 20(2):207–219, 2023. doi: 10.1007/ s11633-022-1387-3. URL htt… — [doi:10.1007/s11633-022-1387-3](https://doi.org/10.1007/s11633-022-1387-3) find
  16. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12):248:1–248:38, 2023. doi: 10.1145/3571730. URL https://doi.org/10.1145/ 3571730. our card
  17. Oliver P John, Sanjay Srivastava, et al. The big-five trait taxonomy: History, measurement, and theoretical perspectives. 1999. find
  18. Scott Prevost Justine Cassell, Joseph Sullivan and Elizabeth F. Churchill. Embodied conversational agents. In MIT Press. 2000. find
  19. Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. Towards emotional support dialog systems. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistic… our card
  20. Abraham Maslow. A theory of human motivation. Lulu. com, 1974. find
  21. Clifford Nass and Youngme Moon. Machines and mindlessness: Social responses to computers. Journal of social issues, 56(1):81–103, 2000. find
  22. Clifford Nass, Jonathan Steuer, and Ellen R Tauber. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 72–78, 1994. find
  23. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. ↗ our card
  24. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Train… our card
  25. Joon Sung Park, Joseph C. O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Sean Follmer, Jeff Han, Jürgen Steimle, and Nathalie Henry Riche, editors, Proceedings of the 36th Annual ACM Symposium on… find
  26. ACM, 2023. doi: 10.1145/3586183.3606763. URL https://doi.org/10.1145/3586183.3606763. — [doi:10.1145/3586183.3606763](https://doi.org/10.1145/3586183.3606763) ↗
  27. Martin J Pickering and Simon Garrod. Toward a mechanistic psychology of dialogue. Behavioral and brain sciences, 27(2):169–190, 2004. find
  28. Byron Reeves and Clifford Nass. The media equation: How people treat computers, television, and new media like real people. Cambridge, UK, 10(10), 1996. find
  29. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. Recipes for building an open-domain chatbot. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the Eur… our card
  30. Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. Blenderbot 3: a deployed conversational agent… our card
  31. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzho… our card
  32. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302… our card
  33. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  34. Ronald Wardhaugh and Janet M Fuller. An introduction to sociolinguistics. John Wiley & Sons, 2021. find
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 9d5609613524ecf4f15af0f7b31abca4-Abstr… ↗ our card
  36. Joseph Weizenbaum. Eliza—a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36–45, 1966. find
  37. Jifan Yu, Xiaohan Zhang, Yifan Xu, Xuanyu Lei, Xinyu Guan, Jing Zhang, Lei Hou, Juanzi Li, and Jie Tang. XDAI: A tuning-free framework for exploiting pre-trained language models in knowledge grounded dialogue generation. In Aidong Zhang and Huzefa Rangwala, editors, KDD ’22: The 28th ACM SIGKDD Conf… find
  38. ACM, 2022. doi: 10.1145/3534678.3539135. URL https://doi.org/10.1145/3534678.3539135. — [doi:10.1145/3534678.3539135](https://doi.org/10.1145/3534678.3539135) ↗
  39. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022. our card
  40. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, M… our card
  41. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. CoRR, abs/2303.17568, 2023. doi: 10.48550/ARXIV.2303.17568.… our card
  42. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification. CoRR, abs/2308.07921, 2023a. doi: 10.48550/arXiv.2308.07921. UR… our card
  43. Hao Zhou, Pei Ke, Zheng Zhang, Yuxian Gu, Yinhe Zheng, Chujie Zheng, Yida Wang, Chen Henry Wu, Hao Sun, Xiaocong Yang, Bosi Wen, Xiaoyan Zhu, Minlie Huang, and Jie Tang. EVA: an open-domain chinese dialogue system with large-scale generative pre-training. CoRR, abs/2108.01547, 2021. URL https://arxi… — [arXiv:2108.01547](https://arxiv.org/abs/2108.01547) find
  44. Jinfeng Zhou, Zhuang Chen, Bo Wang, and Minlie Huang. Facilitating multi-turn emotional support conversation with positive emotion elicitation: A reinforcement learning approach. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Associa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2023-characterglm-customizing-chinese-conversational-ai/