Elamur Research⌕
☰
home / papers / 2018 / Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning

Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning

Paper · 2018 · Pararth Shah, Dilek Hakkani-Tür, Bing Liu et al.

Summary

The paper addresses a practical obstacle for task-oriented conversational agents. End-to-end neural models need large annotated dialogue corpora, which are costly to build for every new domain. Wizard-of-Oz collection also often misses dialogue flows a developer expects the agent to handle.

The authors propose a four-step recipe built on Machines Talking To Machines (M2M):

  1. A task-independent user simulator and a rule-based system agent, given only a slot schema and a database API, converse with each other. This produces annotated dialogue "outlines" written in template language.
  2. Crowd workers rewrite each template turn into natural, context-aware language.
  3. A neural agent is trained on the resulting fully annotated corpus.
  4. The deployed agent keeps improving through on-line reinforcement learning from user feedback.

The released datasets show more lexical and dialogue-flow diversity than DSTC2. A model trained with supervised learning and then refined with RL received better human ratings than the supervised-only model, both with simulated and with real users. The approach matters because it turns a developer's hand-written policy into a trainable model that is safe to deploy from day one.

Key points

  • Released datasets: Sim-M (movie ticket booking) and Sim-R (restaurant table reservation), about 3000 dialogues in total, available in the google-research-datasets/simulated-dialogue repository.
  • Self-play components:
    • an agenda-based user simulator whose behavior is governed by two inputs, the goal the user is pursuing and the user's profile;
    • a handcrafted finite state machine system agent that moves through fixed sub-dialogues: gather preferences, query the API, offer entities, allow changes or questions, complete the transaction.
  • Annotation scheme: dialogue acts follow the Cambridge dialogue act schema, and each outline is rewritten by K > 1 crowd workers.
  • Diversity comparison: Sim-R (1611 training dialogues) was compared with DSTC2 (1116). It had higher unique unigram and bigram ratios and higher outline diversity, measured for sub-dialogues of length 1, 3 and 5 and for full dialogues.
  • Model evaluation:
    • The model (Liu et al. 2017) was trained with supervised learning on Sim-M, then with RL for 10K simulated episodes.
    • Each system was evaluated over 100 episodes, with turns rated 1–5 by three judges.
    • SL+RL beat SL alone.
    • Ratings were lower with real users than with the simulator.

Interesting ideas

  • I1. Separate semantics from wording. Generate the semantic skeleton of each dialogue automatically, then let humans supply only the surface language. Coverage of flows can then be guaranteed by design, and annotations come for free, which helps any low-resource domain or language.
  • I2. Ask crowd workers to paraphrase, not invent. Rewriting a given template is a well-scoped job that yields cleaner data than open-ended dialogue writing. It is a useful recipe for building corpora with modest budgets, for example for Ukrainian.
  • I3. Use rewriting to add realistic phenomena. Showing the whole dialogue context while workers rewrite lets them add coreference and lexical entrainment naturally. These are features that templates alone cannot produce.
  • I4. Measure diversity of flows, not just words. Outline diversity is the share of unique act sequences at several sub-dialogue lengths. It is a simple, reusable metric for checking whether a dialogue dataset covers varied interaction patterns.
  • I5. Distill a rule system into a neural model, then refine it with RL. Converting a hand-written policy into a differentiable model gives a guaranteed baseline at launch. The model can then keep learning from live feedback, a pattern applicable wherever rule-based systems already exist.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 401 works

Cite

Cite as Pararth Shah, Dilek Hakkani-Tür, Bing Liu and Gokhan Tür. 2018. Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-3006

@inproceedings{shah2018bootstrapping,
  title = {{Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning}},
  author = {Pararth Shah, Dilek Hakkani-Tür, Bing Liu and Gokhan Tür},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-3006},
  url = {https://doi.org/10.18653/v1/n18-3006}
}

References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016b. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pages 1192– 1202.
  2. Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: A corpus for adding memory to goal-oriented dialogue systems. arXiv preprint arXiv:1704.00057 .
  3. Bing Liu and Ian Lane. 2017a. An end-to-end trainable neural network model with belief tracking for task-oriented dialog. In Interspeech. find
  4. Paul Crook, Alex Marin, Vipul Agarwal, Khushboo Aggarwal, Tasos Anastasakos, Ravi Bikkula, Daniel Boies, Asli Celikyilmaz, Senthilkumar Chandramohan, Zhaleh Feizollahi, et al. 2016. Task completion platform: A self-serve multi-domain goal oriented dialogue platform. In Proceedings of the 2016 Confer… find
  5. Bing Liu and Ian Lane. 2017b. Iterative policy learning in end-to-end trainable task-oriented neural dialog models. In Proceedings of IEEE ASRU. find
  6. Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2017. End-to-end optimization of task-oriented dialogue model with deep reinforcement learning. In NIPS Workshop on Conversational AI. find
  7. Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of oz studieswhy and how. Knowledge-based systems 6(4):258–266. find
  8. Ryan Lowe, Nissan Pow, Iulian Vlad Serban, Laurent Charlin, Chia-Wei Liu, and Joelle Pineau. 2017. Training end-to-end dialogue systems with the ubuntu dialogue corpus. Dialogue & Discourse 8(1):31–65. find
  9. Layla El Asri, Romain Laroche, and Olivier Pietquin. 2014. Task completion transfer learning for reward inference. Proc of MLIS . find
  10. Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, and Kaheer Suleman. 2016. Policy networks with two-stage training for dialogue systems. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. pages 101– 110. find
  11. Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476 .
  12. Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volum…
  13. Milica Gašić, Filip Jurčı́ček, Blaise Thomson, Kai Yu, and Steve Young. 2011. On-line policy optimisation of spoken dialogue systems via live interaction with human subjects. In Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on. IEEE, pages 312–317. find
  14. Matthew Henderson, Blaise Thomson, and Jason Williams. 2013. Dialog state tracking challenge 2 & 3. http://camdial.org/˜mh521/dstc/. ↗
  15. Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI. volume 16, pages 3776–3784.
  16. John E. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman. 2006. Introduction to Automata Theory, Languages, and Computation (3rd Edition). Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA. find
  17. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence.
  18. John F Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS) 2(1):26–41. find
  19. Pararth Shah, Dilek Hakkani-Tür, and Larry Heck. 2016. Interactive reinforcement learning for task-oriented dialogue management. In NIPS Deep Learning for Action and Interaction Workshop. find
  20. Anjishnu Kumar, Arpit Gupta, Julian Chan, Sam Tucker, Bjorn Hoffmeister, and Markus Dreyer. 2017. Just ask: Building an architecture for extensible self-service spoken language understanding. In NIPS Conversational AI Workshop. find
  21. Pararth Shah, Dilek Hakkani-Tür, Gokhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018. Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871 . find
  22. Jiwei Li, Alexander H Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. 2016a. Dialogue learning with human-in-the-loop. arXiv preprint arXiv:1611.09823 . find
  23. Pei-Hao Su, Paweł Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. 2017. Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue. pages 147–157. find
  24. Pei-Hao Su, Milica Gasic, Nikola Mrkšić, Lina M Rojas Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016a. On-line active reward learning for policy optimisation in spoken dialogue systems. In Proceedings of the 54th Annual Meeting of the Association for Computational Li… find
  25. Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016b. Continuously learning neural dialogue management. arXiv preprint arXiv:1606.02689 . find
  26. Pei-Hao Su, David Vandyke, Milica Gašı́c, Dongho Kim, Nikola Mrkšı́c, Tsung-Hsien Wen, and Steve Young. 2015. Learning from real users: Rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems. In Proceedings of the Annual Conference of the International S… find
  27. Stefan Ultes, Lina M Rojas Barahona, Pei-Hao Su, David Vandyke, Dongho Kim, Inigo Casanueva, Paweł Budzianowski, Nikola Mrkšić, Tsung-Hsien Wen, Milica Gasic, et al. 2017. Pydial: A multi-domain statistical dialogue system toolkit. Proceedings of ACL 2017, System Demonstrations pages 73– 78. find
  28. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869 .
  29. Yushi Wang, Jonathan Berant, Percy Liang, et al. 2015. Building a semantic parser overnight. ACL . find
  30. Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. ACL . *Figure 5: Contextual rewrite task interface for paraphrasing a dialogue outline with natural… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shah-2018-bootstrapping-neural-conversational-agent-dialogue/