Elamur Research⌕
☰
home / papers / 2019 / Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models

Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models

Paper · 2019 · Tiancheng Zhao, Kaige Xie and Maxine Eskenazi

Summary

Applying reinforcement learning to end-to-end dialog agents is hard because the action space is usually either a set of handcrafted dialog acts, which does not scale to complex domains, or the full output vocabulary. Treating every generated word as an action, as word-level RL does, tends to corrupt the language a model produces and makes credit assignment slow over long dialogs. The authors propose Latent Action Reinforcement Learning (LaRL). A latent variable z sits between the dialog context and the response, and policy gradients are applied to z rather than to individual words. The decoder is left unchanged during RL, so strategy is separated from surface generation. They compare Gaussian and categorical latent actions, two pre-training objectives based on stochastic variational inference (a full ELBO and a regularized "Lite" variant), and an attention fusion mechanism for discrete codes. On DealOrNoDeal and MultiWoz, LaRL beats word-level policy gradient baselines without degrading language quality. It also sets a new success-rate state of the art on MultiWoz. This makes the approach a promising basis for unsupervised action abstraction in dialog agents.

Key points

  • Shorter RL horizon: LaRL factorizes generation as p(x|c) = p(x|z)p(z|c) and runs REINFORCE only on the encoder. This cuts the horizon from T·U word steps to T turns.
  • MultiWoz result: The method gains 18.2% absolute over the previous state of the art. Word-level RL with RL:SL=off learned to spam entity tokens to inflate the success metric, whereas LiteAttnCat produced fluent, informative replies.
  • Categorical beats Gaussian: Categorical latent actions (M independent K-way variables, giving K^M combinations) consistently outperformed Gaussian ones. REINFORCE on Gaussian latents was unstable and often diverged.
  • Lite objective and regularization: The Lite objective outperformed the full ELBO for pre-training, which the authors attribute to exposure bias in the latent space. Regularizing the latent space with β = 0.01 instead of 0 was needed for sufficient exploration during RL.
  • Resources: Data and code are released at github.com/snakeztc/NeuralDialog-LaRL. The paper also introduces LCR curves for analyzing the trade-off between reward and language quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓Shanghai Jiao Tong UniversityShanghai · China63 works · 15,047 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Dialogue systems · 401 works

Cite

Cite as Tiancheng Zhao, Kaige Xie and Maxine Eskenazi. 2019. Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1123

@inproceedings{zhao2019rethinking,
  title = {{Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models}},
  author = {Tiancheng Zhao, Kaige Xie and Maxine Eskenazi},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1123},
  url = {https://doi.org/10.18653/v1/n19-1123}
}

References

48 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P… find
  2. Kris Cao and Stephen Clark. 2017. Latent variable dialogue models and their diversity. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 182–187. find
  3. Yun-Nung Chen, William Yang Wang, and Alexander I Rudnicky. 2013. Unsupervised induction and filling of semantic slots for spoken dialogue systems using frame-semantic parsing. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 120–125. IEEE. find
  4. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  5. Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2970–2979. IEEE. find
  6. Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, and Li Deng. 2017. Towards end-to-end reinforcement learning of dialogue agents for information access. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… our card
  7. Milica Gasic and Steve Young. 2014. Gaussian processes for pomdp-based dialogue manager optimization. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22(1):28–40. find
  8. Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. 2004. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(Nov):1471–1530. find
  9. He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2333–2343. our card
  10. Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 292– 299. our card
  11. Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144. our card
  12. Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237–285. find
  13. Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114. our card
  14. Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. 2017. Natural language does not emerge nat-urallyin multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2962–2967. find
  15. John Langford and Tong Zhang. 2008. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in neural information processing systems, pages 817–824. find
  16. Sungjin Lee. 2013. Structured discriminative model for dialog state tracking. In Proceedings of the SIG-DIAL 2013 Conference, pages 442–451. find
  17. Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443–2453. our card
  18. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192– 1202. our card
  19. Bing Liu and Ian Lane. 2017. An end-to-end trainable neural network model with belief tracking for task-oriented dialog. find
  20. Proc. Interspeech 2017, pages 2506–2510. find
  21. Marilyn A. Walker. 2000. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research, pages 387–416. find
  22. Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421. our card
  23. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562. find
  24. Igor Mordatch and Pieter Abbeel. 2017. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv:1703.04908. find
  25. Tsung-Hsien Wen, Yishu Miao, Phil Blunsom, and Steve Young. 2017. Latent intention dialogue models. In International Conference on Machine Learning, pages 3732–3741. find
  26. Antoine Raux, Brian Langner, Dan Bohus, Alan W Black, and Maxine Eskenazi. 2005. Lets go public! taking a spoken dialog system to the real world. In in Proc. of Interspeech 2005. Citeseer. find
  27. Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. Towards universal dialogue state tracking. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2780– 2786. our card
  28. Iulian V Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, Sarath Chandar, Nan Rosemary Ke, et al. 2017a. A deep reinforcement learning chatbot. arXiv preprint arXiv:1709.02349. find
  29. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16). our card
  30. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017b. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence. our card
  31. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the 2015 Conference of the North American Chapte… our card
  32. Pei-Hao Su, Paweł Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. 2017. Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 147–157. find
  33. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869. our card
  34. Jason D Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1,… find
  35. Jason D Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393–422. find
  36. Jason D Williams and Geoffrey Zweig. 2016. End-to-end lstm-based dialog control optimized with supervised and reinforcement learning. arXiv preprint arXiv:1606.01269. — [arXiv:1606.01269](https://arxiv.org/abs/1606.01269) find
  37. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256. find
  38. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 14… our card
  39. Denis Yarats and Mike Lewis. 2017. Hierarchical text generation and planning for strategic dialogue. arXiv preprint arXiv:1712.05846. — [arXiv:1712.05846](https://arxiv.org/abs/1712.05846) find
  40. Stephanie Young, Jost Schatzmann, Karl Weilhammer, and Hui Ye. 2007. The hidden information state approach to dialog management. In Acoustics, Speech and Signal Processing, 2007. find
  41. ICASSP 2007. IEEE International Conference on, volume 4, pages IV– 149. IEEE. find
  42. Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160–1179. find
  43. Steve J Young. 2006. Using pomdps for dialog management. In SLT, pages 8–13. find
  44. Scenario 1 1 book, 1 hat and 3 balls available. Value Function Agent: book=1, hat=6, ball=1. User: book=1, hat=6, ball=1. Dialog Agent: I’d like the hat and 2 balls. User: Deal. Agent: SELECTION Outcome: Agreement Agent: 1 hat and 2 balls; User: 1 book and 1 ball Agent reward = 8; User reward = 2 Sc… find
  45. Tiancheng Zhao and Maxine Eskenazi. 2016. Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning. In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, page 1. find
  46. Tiancheng Zhao and Maxine Eskenazi. 2018. Zero-shot dialog generation with cross-domain latent actions. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 1–10. find
  47. Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018. Unsupervised discrete sentence representation learning for interpretable neural dialog generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1. find
  48. Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 654–664. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2019-rethinking-action-spaces-reinforcement-learning/