Elamur Research⌕
☰
home / papers / 2023 / Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with Characters

Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with Characters

Paper · 2023 · Nuo Chen, Yan Wang, Haiyun Jiang et al.

Summary

Chat-oriented LLMs such as ChatGPT and GPT-4 can hold open-domain conversations, but they struggle to stay in character as a specific fictional person. They mix in real-world knowledge, confuse overlapping relationships, and ignore how characters change as a story unfolds. To address this, the authors build the Harry Potter Dialogue (HPD) dataset. It collects every dialogue session involving Harry from the novels, in both English and Chinese. Each session carries manual annotations for scene, speakers, relations to Harry and speaker attributes. These labels are tied to the point in the storyline, so they can differ from one session to the next. The test set adds negative candidate responses, which allows both generation-based and retrieval-based evaluation. LLMs were benchmarked with fine-tuning and in-context learning. HPD background information helps models give more Harry-like replies, yet human-written responses remain clearly better. HPD therefore offers a benchmark for character alignment, a task with substantial unsolved difficulty.

Key points

  • Data: HPD contains 1042 training sessions, each with one positive response. It has 149 test sessions, each with 1–3 human-written positive responses and about 9 negative ones. The test set was specifically designed to limit knowledge leakage from pre-training data.
  • Annotation: Four dedicated Harry Potter fans did the labelling, with no crowd-sourcing. Each speaker is labelled with 12 relation types (relative to Harry) and 13 attribute types. These labels change with the book and chapter, unlike static persona datasets such as PeDialog or Focus.
  • Bilingual design: The English and Chinese versions both come from the original novels rather than machine translation. The paper reports only English results.
  • Human evaluation: Annotators compared Per-ChatGPT outputs with human-written answers. Per-ChatGPT matched or beat humans in under 30% of cases on every metric.
  • Release: Because of copyright, the authors release only line and page numbers plus an extraction script, for non-commercial use. They release their own attribute and relation annotations separately.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China58 works · 23,058 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China7 works · 1,026 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 401 works

Cite

Cite as Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang and Jia Li. 2023. Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with Characters. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.570

@inproceedings{chen2023large,
  title = {{Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with Characters}},
  author = {Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang and Jia Li},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.570},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.570}
}

References

30 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…
  2. Nuo Chen, Hongguang Li, Yinan Bao, Junqing He, Xinshi Lin, Qi Yang, Jianfeng Liu, Ruyi Gan, Jiaxing Zhang, Baoyuan Wang, et al. 2023. Orca: A few-shot benchmark for chinese conversational machine reading comprehension. arXiv preprint arXiv:2302.13619. find
  3. Nuo Chen, Chenyu You, and Yuexian Zou. 2021. Self-supervised dialogue learning for spoken conversational question answering. In Interspeech, pages 231–235. ISCA. find
  4. Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In CMCL@ACL, pages 76–87. Association for Computational Linguistics.
  5. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In ICLR (Poster). OpenReview.net. find
  6. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233.
  7. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  8. Yoonna Jang, Jungwoo Lim, Yuna Hur, Dongsuk Oh, Suhyune Son, Yeonsoo Lee, Dong-Hoon Shin, Seungryong Kim, and Heuiseok Lim. 2021. Call for customized conversation: Customized conversation grounding persona and knowledge. CoRR, abs/2112.08619. — [arXiv:2112.08619](https://arxiv.org/abs/2112.08619) find
  9. Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. 2023. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint arXiv:2303.05453. find
  10. Satwik Kottur, Xiaoyu Wang, and Vítor Carvalho. 2017. Exploring personalized neural conversational models. In IJCAI, pages 3728–3734. ijcai.org. find
  11. Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and William B. Dolan. 2016. A persona-based neural conversation model. In ACL (1). The Association for Computer Linguistics.
  12. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  13. Yongkang Liu, Shi Feng, Daling Wang, Yifei Zhang, and Hinrich Schütze. 2023. Evaluate what you can’t evaluate: Unassessable generated responses quality. arXiv preprint arXiv:2305.14658. — [arXiv:2305.14658](https://arxiv.org/abs/2305.14658) find
  14. OpenAI. 2023. Gpt-4 technical report.
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.
  16. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277.
  17. Hongjin Qian, Xiaohe Li, Hanxun Zhong, Yu Guo, Yueyuan Ma, Yutao Zhu, Zhanliang Liu, Zhicheng Dou, and Ji-Rong Wen. 2021. Pchatbot: A large-scale dataset for personalized chatbot. In SIGIR, pages 2470–2477. ACM. find
  18. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2023. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304.11406.
  19. Haoyu Song, Yan Wang, Weinan Zhang, Zhengyu Zhao, Ting Liu, and Xiaojiang Liu. 2020. Profile consistency identification for open-domain dialogue agents. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6651–6662. find
  20. Chenyu You, Nuo Chen, and Yuexian Zou. 2021. Self-supervised contrastive cross-modality representation learning for spoken question answering. In EMNLP (Findings), pages 28–39. Association for Computational Linguistics. find
  21. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
  22. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In ACL (1), pages 2204–2213. Association for Computational Linguistics.
  23. Wei-Nan Zhang, Qingfu Zhu, Yifa Wang, Yanyan Zhao, and Ting Liu. 2019. Neural personalized response generation as domain adaptation. World Wide Web, 22(4):1427–1446. find
  24. Yinhe Zheng, Guanyi Chen, Minlie Huang, Song Liu, and Xuan Zhu. 2019. Personalized dialogue generation with diversified traits. CoRR, abs/1901.09672. — [arXiv:1901.09672](https://arxiv.org/abs/1901.09672) find
  25. Yinhe Zheng, Rongsheng Zhang, Minlie Huang, and Xiaoxi Mao. 2020. A pre-training based personalized dialogue generation model with persona-sparse data. In AAAI, pages 9693–9700. AAAI Press. find
  26. Tengtao Song, Nuo Chen, Ji Jiang, Zhihong Zhu, and Yuexian Zou. 2023. Improving retrieval-based dialogue system via syntax-informed attention. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  27. Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. CoRR, abs/1901.08149. find
  28. Chen Xu, Piji Li, Wei Wang, Haoran Yang, Siyun Wang, and Chuangbai Xiao. 2022. Cosplay: Concept set guided personalized dialogue generation across both party personas. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 201– 211. find
  29. Zhengzhe Yang and Jinho D. Choi. 2019. Friendsqa: Open-domain question answering on TV show transcripts. In SIGdial, pages 188–197. Association for Computational Linguistics. find
  30. Chenyu You, Nuo Chen, Fenglin Liu, Dongchao Yang, and Yuexian Zou. 2020. Towards data distillation for end-to-end spoken conversational question answering. CoRR, abs/2010.08923. — [arXiv:2010.08923](https://arxiv.org/abs/2010.08923) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-large-language-models-meet-harry/