Elamur Research⌕
☰
home / papers / 2024 / Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

Paper · 2024 · Keming Lu, Bowen Yu, Chang Zhou et al.

Summary

Open-source role-play chatbots are usually trained on dialogues distilled from proprietary systems such as GPT-4. This requires a stronger teacher, tends to increase hallucination, and brings licensing constraints. The authors argue that an LLM already holds knowledge of countless characters from pre-training. They propose DITTO, a self-alignment recipe that needs no external teacher.

DITTO works in two stages:

  • Simulation: an instruction-following model receives a character's attributes and profile collected from Wikipedia. It generates user queries and in-character replies, treating the task like reading comprehension.
  • Fine-tuning: the model is trained to give those same replies when shown only a short character description. This forces it to draw on its internal knowledge.

The process yields WikiRole, a multilingual self-generated dataset covering 4,000 characters. The authors also build an automatic, LLM-judged benchmark that checks identity consistency, role-specific factual accuracy, and refusal of out-of-scope questions.

Applied to Qwen-Chat models, DITTO beats all open-source role-play baselines and approaches proprietary chatbots. Cross-supervision experiments show that conversational style transfers easily even from smaller models, whereas role knowledge is limited by the base model itself.

Key points

  • WikiRole covers 4,000 Wikipedia characters, about ten times more roles than earlier public role-play datasets. Resources are released at github.com/OFA-Sys/Ditto.
  • DITTO was applied to four Qwen-Chat sizes (1.8B, 7B, 14B, 72B). The 72B version reaches about 90% on role identity consistency and is roughly on par with GPT-3.5-Turbo. It still trails GPT-4 and Qwen-Max on role-related knowledge.
  • The benchmark reduces evaluation to judge-answerable questions:
    • choosing the portrayed character from four options;
    • checking the dialogue against gold knowledge;
    • testing whether the model admits ignorance, e.g., when Harry Potter is asked about quicksort.
  • Evaluation also uses the role-play subset of MT-Bench.
  • In cross-supervision, identity consistency improves after imitation even when the teacher is weaker. Knowledge and rejection scores, by contrast, are capped by the seed model.
  • Weak-to-strong generalization appears on knowledge metrics. Qwen-72B-Chat tuned on simulations from Qwen-1.8B-Chat matches self-aligned Qwen-7B-Chat.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China64 works · 14,113 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Language models & pretraining · 1,343 works

Cite

Cite as Keming Lu, Bowen Yu, Chang Zhou and Jingren Zhou. 2024. Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.423

@inproceedings{lu2024large,
  title = {{Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment}},
  author = {Keming Lu, Bowen Yu, Chang Zhou and Jingren Zhou},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.423},
  url = {https://doi.org/10.18653/v1/2024.acl-long.423}
}

References

23 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. our card
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. our card
  3. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. 2023. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. our card
  4. Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang, and Jia Li. 2023. Large language models meet harry potter: A dataset for aligning dialogue agents with characters. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8506–8520. our card
  5. Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023b. \# InsTag: Instruction tagging for analyzing supervised fine-tuning of large language models. arXiv e-prints, pages arXiv–2308. find
  6. Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124. find
  7. Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. Role play with large language models. Nature, pages 1–6. our card
  8. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-llm: A trainable agent for role-playing. arXiv preprint arXiv:2310.10158. our card
  9. Meiling Tao, Xuechen Liang, Tianyu Shi, Lei Yu, and Yiting Xie. 2023. Rolecraft-glm: Advancing personalized role-playing in large language models. find
  10. Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717. our card
  11. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. find
  12. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  13. Quan Tu, Chuanqi Chen, Jinpeng Li, Yanran Li, Shuo Shang, Dongyan Zhao, Ran Wang, and Rui Yan. 2023. Characterchat: Learning towards conversational ai with personalized social support. arXiv preprint arXiv:2308.10278. — [arXiv:2308.10278](https://arxiv.org/abs/2308.10278) find
  14. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  15. Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023a. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235. our card
  16. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023b. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966. our card
  17. Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi MI, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, et al. 2023a. Chatharuhi: Reviving anime character in reality via large language model. arXiv preprint arXiv:2308.09597. our card
  18. Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al. 2023c. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746. our card
  19. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. 2023b. Self-alignment with instruction back-translation. arXiv preprint arXiv:2308.06259. our card
  20. Mengge Xue, Bowen Yu, Zhenyu Zhang, Tingwen Liu, Yue Zhang, and Bin Wang. 2020. Coarse-to-fine pre-training for named entity recognition. arXiv preprint arXiv:2010.08210. find
  21. Keming Lu, Xiaoman Pan, Kaiqiang Song, Hongming Zhang, Dong Yu, and Jianshu Chen. 2023a. Pivoine: Instruction tuning for open-world entity profiling. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15108–15127. find
  22. Jifan Yu, Xiaohan Zhang, Yifan Xu, Xuanyu Lei, Xinyu Guan, Jing Zhang, Lei Hou, Juanzi Li, and Jie Tang. 2022. Xdai: A tuning-free framework for exploiting pre-trained language models in knowledge grounded dialogue generation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery a… find
  23. Query for collecting virtual characters Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2024-large-language-models-are-superpositions/