Elamur Research
☰
home / papers / 2020 / Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills

Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.183.pdf

Summary

Open-domain chatbots are usually trained and tested on one quality at a time: persona-based small talk (ConvAI2), knowledge grounding (Wizard of Wikipedia) or empathy (EmpatheticDialogues). This paper asks whether those separately learned abilities can be merged so that one agent switches between them, or mixes them, within a single dialogue. To test this, the authors crowdsourced BlendedSkillTalk, an English dataset of about 5k conversations. In each conversation, one worker was shown optional reply suggestions from three single-skill poly-encoder models.

They then compared several ways of combining skills:

  • picking a skill model at random;
  • a two-stage system in which a classifier routes each turn to a single-skill expert;
  • multi-task training on all three datasets;
  • fine-tuning on the new blended data.

Multi-task training beat every single-skill baseline, and adding BST fine-tuning gave the best results on both automatic and human evaluations. A compact multi-task model matched a two-stage system more than three times its size. The work offers a benchmark and practical recipes for building agents that combine several conversational skills.

Key points

  • BlendedSkillTalk (BST) data: 4,819 train, 1,009 validation and 980 test conversations from 2,679 workers. The worker pools for the three splits do not overlap, and train conversations average 11.2 utterances. The dataset is distributed through ParlAI.
  • Collection setup:
    • Each conversation is seeded with an utterance pair from ConvAI2, WoW or ED, plus personas.
    • Guided workers adopted model suggestions in 20.5% of their turns.
    • By the dataset classifier, 47.8% of train dialogues show all three skill modes, 43.2% show two and 9.1% show one.
  • BST test results:
    • Multi-Task Single-Skills scored 80.1 zero-shot, the best zero-shot result.
    • It rose to 83.8 after BST fine-tuning, the best result overall.
    • Single-skill models scored 67.5–76.8 zero-shot and 79.4–81.7 after fine-tuning.
  • Human ratings (overall quality): MT Single-Skills + BST fine-tuning scored 3.6 and MT Two-Stage 3.5. The fine-tuned model uses under a third of the two-stage model's parameters.
  • Random-Skill baseline: it was rated well on the personal axis but weaker on knowledge and empathy. This suggests those two skills must fit the conversational context to work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

118 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 111 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.183

@inproceedings{smith2020can,
  title = {{Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills}},
  author = {Smith, Eric Michael and Williamson, Mary and Shuster, Kurt and Weston, Jason and Boureau, Y-Lan},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.183},
  url = {https://doi.org/10.18653/v1/2020.acl-main.183}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  2. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019a. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Na… find our card
  3. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2020. The second conversational… find our card
  4. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019b. Wizard of wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations. find
  5. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf… find
  6. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Real-time inference in multi-sentence tasks with deep pretrained transformers. arXiv preprint arXiv:1905.01969. — [arXiv:1905.01969](https://arxiv.org/abs/1905.01969) find
  7. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858. find our card
  8. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995, Taipei, Taiwan. Asian Federati… find our card
  9. Zhaojiang Lin, Andrea Madotto, Jamin Shin, Peng Xu, and Pascale Fung. 2019. MoEL: Mixture of empathetic listeners. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages… find
  10. Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2775–2779, Brussels, Belgium. Association for Computational Linguistics… find
  11. Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018. Towards exploiting background knowledge for building conversation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2322–2332, Brussels, Belgium. Association for Comput… find
  12. Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, and Jianfeng Gao. 2019. Conversing by reading: Contentful neural conversation with on-demand machine reading. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages… find
  13. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  14. Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, et al. 2018. Conversational ai: The science behind the alexa prize. arXiv preprint arXiv:1801.03604. — [arXiv:1801.03604](https://arxiv.org/abs/1801.03604) find
  15. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Com… find
  16. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204– 2213, Melbo… find our card
  17. Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020. The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics, 46(1):53–93. find
  18. Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79–84, Copenhagen… find our card