Elamur Research
☰
home / papers / 2018 / Personalizing Dialogue Agents: I have a dog, do you have pets too?

Personalizing Dialogue Agents: I have a dog, do you have pets too?

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1205.pdf

Summary

Open-domain chit-chat bots often feel bland. Their personality shifts from turn to turn, they forget earlier context, and they fall back on vague replies. The authors address this by giving a dialogue agent an explicit text persona, a few short sentences describing who it is. They also release PERSONA-CHAT, a crowdsourced English corpus built around such profiles. Pairs of Mechanical Turk workers each got a random persona and chatted to get to know each other. A separate group of workers then rewrote every persona in new wording, so models cannot succeed through simple word overlap. For next-utterance prediction, the paper compares IR baselines, Seq2Seq generators and ranking models, including Memory Networks and a Key-Value Profile Memory network. Giving a model its own persona improves prediction in both the generative and the ranking setting. Human judges found persona-conditioned agents more consistent but not more engaging. Every model trained on PERSONA-CHAT was rated above models trained on OpenSubtitles or Twitter. This makes the corpus a key benchmark for consistent, persona-grounded chatbots.

Key points

  • PERSONA-CHAT has 1,155 personas of at least 5 sentences each, with 100 held out for validation and 100 for test. It contains 10,981 dialogues and 164,356 utterances, and is released with code and models in ParlAI.
  • Each persona also has a revised version: crowdworkers rewrote it as rephrasings, generalizations or specializations. This makes the task harder and gives a semantic-similarity resource.
  • Conditioning on the agent's own persona helps next-utterance prediction. Conditioning on the partner's persona has much less effect, since speakers mostly talk about themselves.
  • In human evaluation, workers chatted with models without knowing it and rated fluency, engagingness and consistency on a 1–5 scale. PERSONA-CHAT models beat models trained on OpenSubtitles (2009 and 2018 versions) and Twitter.
  • In profile prediction, a speaker's persona could be identified from what they said with 94.3% accuracy. It was 23% when using only the KV Profile Memory model's utterances, and accuracy rose as dialogues got longer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

686 papers of our library cite it (of the 140673 whose bibliographies we read); 32 of them have a page here, 654 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1205 · arXiv 1801.07243

@inproceedings{zhang2018personalizing,
  title = {{Personalizing Dialogue Agents: I have a dog, do you have pets too?}},
  author = {Zhang, Saizheng and Dinan, Emily and Urbanek, Jack and Szlam, Arthur and Kiela, Douwe and Weston, Jason},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1205},
  eprint = {1801.07243},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p18-1205}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antoine Bordes and Jason Weston. 2016. Learning end-to-end goal-oriented dialog. arXiv preprint arXiv:1605.07683. find
  2. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051. find our card
  3. Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam, and Jason Weston. 2015. Evaluating prerequisite qualities for learning end-to-end dialog systems. arXiv preprint arXiv:1511.06931. find
  4. Robin IM Dunbar, Anna Marriott, and Neil DC Duncan. 1997. Human conversational behavior. Human nature, 8(3):231–246. find
  5. Chaitanya K Joshi, Fei Mi, and Boi Faltings. 2017. Personalization in goal-oriented dialog. arXiv preprint arXiv:1706.07503. — [arXiv:1706.07503](https://arxiv.org/abs/1706.07503) find
  6. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055. find our card
  7. Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016a. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155. find our card
  8. Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016b. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541. find our card
  9. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023. find
  10. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv preprint arXiv:1506.08909. find our card
  11. JM Lucas, F Fernández, J Salazar, J Ferreiros, and R San Segundo. 2009. Managing speaker identity and user profiles in a spoken dialogue system. Procesamiento del Lenguaje Natural, 43:77–84. find
  12. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126. find
  13. Mor Naaman, Jeffrey Boase, and Chih-Hui Lai. 2010. Is it really about me?: message content in social awareness streams. In Proceedings of the 2010 ACM conference on Computer supported cooperative work, pages 189–192. ACM. find
  14. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  15. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. find our card
  16. Iulian V Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, Sarath Chandar, Nan Rosemary Ke, et al. 2017a. A deep reinforcement learning chatbot. arXiv preprint arXiv:1709.02349. — [arXiv:1709.02349](https://arxiv.org/abs/1709.02349) find
  17. Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2015. A survey of available corpora for building data-driven dialogue systems. arXiv preprint arXiv:1512.05742. — [arXiv:1512.05742](https://arxiv.org/abs/1512.05742) find
  18. Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2016. Generative deep neural networks for dialogue: A short review. arXiv preprint arXiv:1611.06216. — [arXiv:1611.06216](https://arxiv.org/abs/1611.06216) find
  19. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. 2017b. A hierarchical latent variable encoder-decoder model for generating dialogues. find our card
  20. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. arXiv preprint arXiv:1506.06714. find our card
  21. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440–2448. find
  22. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  23. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869. find our card
  24. Ellen M Voorhees et al. 1999. The trec-8 question answering track report. In Trec, volume 99, pages 77– 82. find
  25. Joseph Weizenbaum. 1966. Elizaa computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36–45. find
  26. Ledell Wu, Adam Fisch, Sumit Chopra, Keith Adams, Antoine Bordes, and Jason Weston. 2017. Starspace: Embed all the things! arXiv preprint arXiv:1709.03856. — [arXiv:1709.03856](https://arxiv.org/abs/1709.03856) find
  27. Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160–1179. find
  28. Steve J Young. 2000. Probabilistic methods in spoken– dialogue systems. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 358(1769):1389–1402. find