Elamur Research⌕
☰
home / papers / 2017 / Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access

Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access

Summary

The paper addresses how dialogue agents that help users find an entity in a Knowledge Base (KB-InfoBots) can be trained end-to-end. Earlier systems built a symbolic query from parsed user input and ran an exact database lookup (a "Hard-KB lookup"). That approach has two problems: it discards the parser's uncertainty, and it is non-differentiable, so the language understanding and the policy must be trained separately. The authors replace it with a "Soft-KB lookup". From the agent's per-slot beliefs, it computes a posterior probability for every row of an entity-centric KB, and this full distribution is passed to the policy network. They compare rule-based, RL and fully neural end-to-end (E2E) agents, trained with an adapted user simulator and also evaluated with real users. Soft lookup lets the reinforcement learner find better policies, with higher success rates and rewards than the hard-lookup agent. The E2E agent learns strongly in simulation but overfits. This matters because it is a step toward dialogue agents that keep adapting online from user feedback.

Key points

  • The KB is treated as a table: rows are head entities, columns are slots (relations), and some cells may be missing. The posterior over rows is derived under two assumptions:
    • slots are independent;
    • for each slot, the user may or may not know the value.
  • Compared agents: Rule-Soft, RL-Soft, RL-Hard and an E2E agent. The baseline is an entropy-minimization strategy that always asks about the slot with maximum entropy among the remaining entries.
  • RL-Soft reached 74% task success with real users and 80% against the simulator, which suggests little overfitting.
  • Both soft-lookup agents beat RL-Hard on success and average reward, though they used more turns. RL-Soft matched Rule-Soft's success in fewer turns.
  • The E2E agent scored best in simulation but did poorly with humans. Its simulator vocabulary was small (3078 words). The simulator was adapted from Li et al. (2016b), and the code is released at github.com/MiuLab/KB-InfoBot.

Interesting ideas

  • I1. Pass a distribution instead of a result list. Feeding a downstream component a probability distribution over all candidates, rather than a few retrieved hits, keeps the uncertainty and makes the pipeline differentiable. This could suit any retrieval-backed neural system.
  • I2. Model missing values explicitly. The KB lookup handles empty cells and the chance that the user does not know an attribute. This is useful for incomplete real-world databases.
  • I3. Stress-test with tunable noise. Raising the temperature of the simulator's NLG softmax makes its replies more generic and noisy. It is a cheap, controllable robustness test for agents trained against simulators.
  • I4. Check simulator results against humans. Comparing performance with the simulator and with real users exposes overfitting to the simulator, as happened with the E2E agent. This check is worth copying whenever a model is trained in a synthetic environment.
  • I5. Deploy in stages. Start with a robust RL-Soft agent, collect interaction data, and gradually switch to a personalized E2E agent trained on that experience. This is a practical recipe for adapting systems to individual users.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Dialogue systems

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1045

@inproceedings{dhingra2017end,
  title = {{Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access}},
  author = {Dhingra, Bhuwan and Li, Lihong and Li, Xiujun and Gao, Jianfeng and Chen, Yun-Nung and Ahmed, Faisal and Deng, Li},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1045},
  url = {https://doi.org/10.18653/v1/p17-1045}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Layla El Asri, Jing He, and Kaheer Suleman. 2016. A sequence-to-sequence model for user simulation in spoken dialogue systems. arXiv preprint arXiv:1607.00070 . — [arXiv:1607.00070](https://arxiv.org/abs/1607.00070) find
  2. Geoffrey Hinton, N Srivastava, and Kevin Swersky. 2012. Lecture 6a overview of mini–batch gradient descent. Coursera Lecture slides https://class.coursera. org/neuralnets-2012-001/lecture,[Online . ↗
  3. Antoine Bordes and Jason Weston. 2016. Learning end-to-end goal-oriented dialog. arXiv preprint arXiv:1605.07683 . find
  4. Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016a. Deep reinforcement learning for dialogue generation. EMNLP . find our card
  5. Yun-Nung Chen, Dilek Hakanni-Tür, Gokhan Tur, Asli Celikyilmaz, Jianfeng Guo, and Li Deng. 2016a. find
  6. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. EMNLP . find our card
  7. Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016b. A user simulator for task-completion dialogues. arXiv preprint arXiv:1612.05688 . — [arXiv:1612.05688](https://arxiv.org/abs/1612.05688) find
  8. Xuijun Li, Yun-Nung Chen, Lihong Li, and Jianfeng Gao. 2017. End-to-end task-completion neural dialogue systems. arXiv preprint arXiv:1703.01008 . find our card
  9. Jason D Williams and Steve Young. 2005. Scaling up POMDPs for dialog management: The “Summary POMDP” method. In IEEE Workshop on Automatic Speech Recognition and Understanding, 2005.. IEEE, pages 177–182. find
  10. Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. find
  11. Interspeech 2016 pages 685–689. find
  12. Jason D Williams and Geoffrey Zweig. 2016. End-to-end lstm-based dialog control optimized with supervised and reinforcement learning. arXiv preprint arXiv:1606.01269 . — [arXiv:1606.01269](https://arxiv.org/abs/1606.01269) find
  13. Sebastian Möller, Roman Englert, Klaus-Peter Engelbrecht, Verena Vanessa Hafner, Anthony Jameson, Antti Oulasvirta, Alexander Raake, and Norbert Reithinger. 2006. Memo: towards automatic usability evaluation of spoken dialogue services by user error simulations. In INTERSPEECH. find
  14. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4):229–256. find
  15. Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007a. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volu… find our card
  16. Ji Wu, Miao Li, and Chin-Hui Lee. 2015. A probabilistic framework for representing dialog systems and entropy-based dialog management through dynamic stochastic state evolution. IEEE/ACM Transactions on Audio, Speech, and Language Processing 23(11):2026–2035. find
  17. Kaisheng Yao, Baolin Peng, Yu Zhang, Dong Yu, Geoffrey Zweig, and Yangyang Shi. 2014. Spoken language understanding using long short-term memory neural networks. In Spoken Language Technology Workshop (SLT), 2014 IEEE. IEEE, pages 189–194. find
  18. Jost Schatzmann, Blaise Thomson, and Steve Young. 2007b. Statistical user simulation with a hidden agenda. Proc SIGDial, Antwerp 273282(9). find
  19. Konrad Scheffler and Steve Young. 2002. Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. In Proceedings of the second international conference on Human Language Technology Research. Morgan Kaufmann Publishers Inc., pages 12–19. find
  20. Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016a. Neural generative question answering. International Joint Conference on Artificial Intelligence . find
  21. Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. 2016b. Neural enquirer: Learning to query tables. International Joint Conference on Artificial Intelligence . find
  22. Amanda Spink, Dietmar Wolfram, Major BJ Jansen, and Tefko Saracevic. 2001. Searching the web: The public and their queries. Journal of the Association for Information Science and Technology 52(3):226– 234. find
  23. Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. POMDP-based statistical spoken dialog systems: A review. Proceedings of the IEEE 101(5):1160–1179. find
  24. Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. Paradise: A framework for evaluating spoken dialogue agents. In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics. Association for Computational Linguistics, pages 271–2… find our card
  25. Wojciech Zaremba and Ilya Sutskever. 2015. Reinforcement learning neural Turing machines-revised. arXiv preprint arXiv:1505.00521 . find
  26. Tiancheng Zhao and Maxine Eskenazi. 2016. Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning. arXiv preprint arXiv:1606.02560 . find
  27. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016a. Conditional generation and snapshot learning in neural dialogue systems. EMNLP . find
  28. Stefan Zwicklbauer, Christin Seifert, and Michael Granitzer. 2013. Do we need entity-centric knowledge bases for entity disambiguation? In Proceedings of the 13th International Conference on Knowledge Management and Knowledge Technologies. ACM, page 4. find
  29. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016b. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562 . find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dhingra-2017-end-end-reinforcement-learning-dialogue/