Elamur Research⌕
☰
home / papers / 2016 / Continuously Learning Neural Dialogue Management

Continuously Learning Neural Dialogue Management

Paper · 2016 · Pei-Hao Su, Milica Gasic, Nikola Mrksic et al.

Summary

Task-oriented spoken dialogue systems need a dialogue manager that picks the next system action. Reinforcement learning alone behaves poorly early in training. Supervised imitation of logged dialogues ignores long-term consequences and may not cover all situations. The authors combine both approaches in one neural policy network that maps the belief state to a full master dialogue action (dialogue act, query slot, offered slots).

The network is first trained by supervision on 720 dialogues from the Cambridge restaurant domain, collected through Amazon Mechanical Turk. The same model, with no change to its architecture, is then refined with policy-gradient RL using episodic Natural Actor-Critic (eNAC) and experience replay. This refinement runs against a simulated user and against paid human callers.

The supervised model already predicts actions accurately. RL raises dialogue success further, most clearly when speech understanding is noisier than in the training data. The work shows that a small corpus can bootstrap a usable policy, which then adapts online to new conditions, in a training scheme similar to AlphaGo's.

Key points

  • Policy network design: one tanh hidden layer of 32 units and about 2,600 weights in total.
    • Two softmax outputs: DiaAct (request, offer, confirm, select, bye) and Query (food, pricerange, area, none).
    • Six sigmoid outputs for the Offer slots.
    • The domain covers about 150 venues, and the model was implemented in Theano.
  • Supervised phase: trained on a 4:1:1 train/validation/test split using Adagrad and early stopping. Weighted F-1 on the test set was:
    • DiaAct: 97.73
    • Query: 87.39
    • Offer: 92.51
  • RL setup: eNAC natural gradient with a replay pool of 2,000 and mini-batches of 32. Exploration ε was set to 0.1. Each dialogue's return was 20 × success minus the number of turns, normalised to [0, 1], with a cap of 30 turns.
  • Simulation results: over 6,000 simulated dialogues, success improved by 1–8% across semantic error rates. The largest gains came at high error rates, where the policy learned to confirm more often.
  • Human evaluation: after 200 RL training dialogues with AMT users, 110 evaluation dialogues per policy were collected. SL+RL beat SL alone:
    • Success: 98.2% vs 94.5%
    • Quality: 4.04 vs 3.97 (on a 0–5 scale)

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom256 works · 123,373 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 654 works

Cite

Cite as Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen and Steve Young. 2016. Continuously Learning Neural Dialogue Management. arXiv:1606.02689.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1606.02689 · arXiv 1606.02689

@misc{su2016continuously,
  title = {{Continuously Learning Neural Dialogue Management}},
  author = {Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen and Steve Young},
  year = {2016},
  doi = {10.48550/arxiv.1606.02689},
  eprint = {1606.02689},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1606.02689}
}

References

31 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shun-Ichi Amari. 1998. Natural gradient works efficiently in learning. Neural computation, 10(2):251–276. find
  2. Antoine Bordes and Jason Weston. 2016. Learning end-to-end goal-oriented dialog. arXiv preprint: 1605.07683, May. find
  3. Heriberto Cuayáhuitl, Simon Keizer, and Oliver Lemon. 2015. Strategic dialogue management via deep reinforcement learning. arXiv preprint arXiv:1511.08099. — [arXiv:1511.08099](https://arxiv.org/abs/1511.08099) find
  4. Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of oz studies: why and how. In Proc of Intelligent user interfaces. find
  5. Lucie Daubigney, Matthieu Geist, Senthilkumar Chandramohan, and Olivier Pietquin. 2014. A comprehensive reinforcement learning framework for dialogue management optimisation. Journal of Selected Topics in Signal Processing, 6(8). find
  6. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  7. Layla El Asri, Romain Laroche, and Olivier Pietquin. 2014. Task completion transfer learning for reward inference. In Proc of MLIS. find
  8. Milica Gašić and Steve Young. 2014. Gaussian processes for pomdp-based dialogue manager optimization. TASLP, 22(1):28–40. find
  9. Milica Gašić, Catherine Breslin, Matthew Henderson, Dongho Kim, Martin Szummer, Blaise Thomson, Pirros Tsiakoulis, and Steve J. Young. 2013. On-line policy optimisation of bayesian spoken dialogue systems via human interaction. In Proc of ICASSP. find
  10. M. Henderson, B. Thomson, and J. Williams. 2014a. The Second Dialog State Tracking Challenge. In Proc of SIGdial. find
  11. M. Henderson, B. Thomson, and S. J. Young. 2014b. Word-based Dialog State Tracking with Recurrent Neural Networks. In Proc of SIGdial.
  12. Filip Jurčı́ček, Blaise Thomson, and Steve Young. 2011. Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as pomdps. ACM TSLP, 7(3):6. find
  13. John F. Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transaction on Information Systems. find
  14. Esther Levin and Roberto Pieraccini. 1997. A stochastic model of computer-human interaction for learning dialogue strategies. Eurospeech. find
  15. Long-Ji Lin. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293–321. find
  16. Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2015. Multi-domain Dialog State Tracking using Recurrent Neural Networks. In Proc of ACL. find
  17. Jan Peters and Stefan Schaal. 2006. Policy gradient methods for robotics. In IEEE RSJ. find
  18. Nicholas Roy, Joelle Pineau, and Sebastian Thrun. 2000. Spoken dialogue management using probabilistic reasoning. In Proc of SigDial. find
  19. Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. 2006. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review, 21(02):97–126. find
  20. John Schulman, Sergey Levine, Philipp Moritz, Michael I Jordan, and Pieter Abbeel. 2015. Trust region policy optimization. Proc of ICML. find
  21. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2015. Hierarchical neural network generative models for movie dialogues. arXiv preprint arXiv:1507.04808.
  22. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484– 489. find
  23. Pei-Hao Su, David Vandyke, Milica Gašić, Dongho Kim, Nikola Mrkšić, Tsung-Hsien Wen, and Steve Young. 2015. Learning from real users: Rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems. In Proc of Interspeech. find
  24. Pei-Hao Su, Milica Gašić, Nikola Mrkšić, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. On-line active reward learning for policy optimisation in spoken dialogue systems. In Proc of ACL. find
  25. Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv eprints, abs/1605.02688.
  26. David Vandyke, Pei-Hao Su, Milica Gašić, Nikola Mrkšić, Tsung-Hsien Wen, and Steve Young. 2015. Multi-domain dialogue success classifiers for policy training. In ASRU. find
  27. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869.
  28. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Associat…
  29. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint: 1604.04562, April.
  30. Jason D. Williams and Steve Young. 2007. Partially observable Markov decision processes for spoken dialog systems. Computer Speech and Language, 21(2):393–422. find
  31. Steve Young, Milica Gašic, Blaise Thomson, and Jason Williams. 2013. Pomdp-based statistical spoken dialogue systems: a review. In Proc of IEEE, volume 99, pages 1–20. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/su-2016-continuously-learning-neural-dialogue-management/