Elamur Research⌕
☰
home / papers / 2008 / Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets

Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets

Paper · 2008 · James Henderson, Oliver Lemon and Kallirroi Georgila

Summary

The paper tackles how to learn a dialogue manager directly from a fixed corpus of human–machine dialogues, without user simulation during training or additional data collection. The setting is hard because Information State Update (ISU) systems describe context with many features. This produces an enormous state space and policy space, and any corpus covers only a small part of either. The authors annotated the COMMUNICATOR flight-booking corpus with user actions and Information States. They then trained a hybrid model. Its reinforcement learning (RL) part optimizes dialogue reward, and its supervised learning (SL) part keeps the policy within regions the data actually covers. Linear function approximation lets the model generalize to unseen states. Policies were evaluated against simulated users built from a separate portion of the corpus. Scoring used filled slots, confirmed slots and dialogue length. The hybrid policies beat pure RL, pure SL and the original hand-crafted COMMUNICATOR systems. This shows that rich, feature-based context models can be optimized from limited initial data.

Key points

  • The task is large: 74 possible system actions, a state space reported as roughly 10^386 states, and very few hand-coded constraints. Training used only about 1,000 dialogues.
  • State frequencies in the data are Zipfian. 61% of system turns occur in states seen only once, which motivates generalizing through function approximation rather than estimating each state separately.
  • The SL probability weights how the RL value estimate is combined with a default value for poorly covered states. Two variants of this hybrid were proposed, and both outperform RL and SL used alone.
  • The best hybrid policy outperforms standard RL by 302% and SL by 1.4% on the automatic metric. It also improves on the average COMMUNICATOR system policy by 10%.
  • Tests with human users (Lemon, Georgila, and Henderson 2006) supported these results, with the learned policy beating a hand-coded system. System actions follow the DATE annotation scheme.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations🎓University of GenevaGeneva · Switzerland29 works · 3,158 citations

Also on the byline: Battelle.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 654 works

Cite

Cite as James Henderson, Oliver Lemon and Kallirroi Georgila. 2008. Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.2008.07-028-r2-05-82

@article{henderson2008hybrid,
  title = {{Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets}},
  author = {James Henderson, Oliver Lemon and Kallirroi Georgila},
  journal = {Computational Linguistics},
  year = {2008},
  doi = {10.1162/coli.2008.07-028-r2-05-82},
  url = {https://doi.org/10.1162/coli.2008.07-028-r2-05-82}
}

References

51 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bohlin, Peter, Robin Cooper, Elisabet Engdahl, and Staffan Larsson. 1999. Information states and dialog move engines. Electronic Transactions in AI, 3(9). Available at www.ep.liu.se/ej/etai/1999/D/. find
  2. Bos, Johan, Ewan Klein, Oliver Lemon, and Tetsushi Oka. 2003. DIPPER: Description and formalisation of an information-state update dialogue system architecture. In Proceedings of the 4th SIGdial Workshop on Discourse and Dialogue, pages 115–124, Sapporo. find
  3. Cheyer, Adam and David Martin. 2001. The open agent architecture. Journal of Autonomous Agents and Multi-Agent Systems, 4(1/2):143–148. find
  4. Denecke, Matthias, Kohji Dohsaka, and Mikio Nakano, 2005. Fast reinforcement learning of dialogue policies using stable function approximation. In K. Y. Su, J. Tsujii, J.-H. find
  5. Lee, and O. Y. Kwong, Natural Language Processing, IJCNLP 2004. Springer, Berlin, pages 1–11. find
  6. Frampton, Matthew and Oliver Lemon. 2005. Reinforcement learning of dialogue strategies using the user’s last dialogue act. In Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems, International Joint Conference on Artificial Intelligence (IJCAI), pages 83–90, Edinb… find
  7. Frampton, Matthew and Oliver Lemon. 2006. Learning more effective dialogue strategies using limited dialogue move features. In Proceedings of the 44th Meeting of the Association for Computational Linguistics, pages 185–192, Sydney. find
  8. Gabsdil, Malte and Oliver Lemon. 2004. Combining acoustic and pragmatic features to predict recognition performance in spoken dialogue systems. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics, pages 344–351, Barcelona.
  9. Georgila, Kallirroi, James Henderson, and Oliver Lemon. 2005. Learning user simulations for Information State Update dialogue systems. In Proceedings of the 9th European Conference on Speech Communication and Technology (Interspeech – Eurospeech), pages 893–896, Lisbon. find
  10. Georgila, Kallirroi, James Henderson, and Oliver Lemon. 2006. User simulation for spoken dialogue systems: Learning and evaluation. In Proceedings of the 9th International Conference on Spoken Language Processing (Interspeech – ICSLP), pages 1065–1068, Pittsburgh, PA. find
  11. Georgila, Kallirroi, Oliver Lemon, and James Henderson. 2005. Automatic annotation of COMMUNICATOR dialogue data for learning dialogue strategies and user simulations. In Proceedings of the Ninth Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL), pages 61–68, Nancy. find
  12. Georgila, Kallirroi, Oliver Lemon, James Henderson, and Johanna Moore. (submitted). Automatic annotation of context and speech acts for dialogue corpora. find
  13. Goddeau, D. and J. Pineau. 2000. Fast reinforcement learning of dialog strategies. In Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pages II–1233–1236, Istanbul. find
  14. Henderson, James, Oliver Lemon, and Kallirroi Georgila. 2005. Hybrid reinforcement/supervised learning for dialogue policies from COMMUNICATOR data. In Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems, International Joint Conference on Artificial Intelligence (I… find
  15. Larsson, Staffan and David Traum. 2000. Information state and dialogue management in the TRINDI Dialogue Move Engine Toolkit. Natural Language Engineering, 6(3–4):323–340. find
  16. Lemon, Oliver, Kallirroi Georgila, and James Henderson. 2006. Evaluating effectiveness and portability of reinforcement learned dialogue strategies with real users: the TALK TownInfo evaluation. In Proceedings of the IEEE/ACL 2006 Workshop on Spoken Language Technology, pages 178–181, Aruba. find
  17. Lemon, Oliver, Kallirroi Georgila, James Henderson, Malte Gabsdil, Ivan Meza-Ruiz, and Steve Young. 2005. Integration of learning and adaptivity with the ISU approach. Technical Report D4.1, TALK Project. find
  18. Lemon, Oliver, Kallirroi Georgila, James Henderson, and Matthew Stuttle. 2006. An ISU dialogue system exhibiting reinforcement learning of dialogue policies: generic slot-filling in the TALK in-car system. In Proceedings of the Demonstrations of EACL, pages 119–122, Trento. find
  19. Lemon, Oliver, Kallirroi Georgila, and Matthew Stuttle. 2005. Showcase exhibiting reinforcement learning for dialogue strategies in the in-car domain. Technical Report D4.2, TALK Project. find
  20. Levin, Esther and Roberto Pieraccini. 1997. A stochastic model of computer-human interaction for learning dialogue strategies. In Proceedings of the 5th European Conference on Speech Communication and Technology (Interspeech – Eurospeech), pages 1883–1886, Rhodes. find
  21. Levin, Esther, Roberto Pieraccini, and Wieland Eckert. 2000. A stochastic model of human-machine interaction for learning dialog strategies. IEEE Transactions on Speech and Audio Processing, 8(1):11–23. find
  22. Pietquin, Olivier. 2004. A Framework for Unsupervised Learning of Dialogue Strategies. Presses Universitaires de Louvain, SIMILAR Collection. find
  23. Pietquin, Olivier and Thierry Dutoit. 2006a. Dynamic Bayesian networks for NLU simulation with application to dialog optimal strategy learning. In Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pages 49–52, Toulouse. find
  24. Pietquin, Olivier and Thierry Dutoit. 2006b. A probabilistic framework for dialog simulation and optimal strategy learning. IEEE Transactions on Speech and Audio Processing, 14(2):589–599. find
  25. Rieser, Verena and Oliver Lemon. 2006a. Cluster-based user simulations for learning dialogue strategies and the SUPER evaluation metric. In Proceedings of the 9th International Conference on Spoken Language Processing (Interspeech – ICSLP), pages 1766–1769, Pittsburgh, PA. find
  26. Rieser, Verena and Oliver Lemon. 2006b. Using logistic regression to initialise reinforcement-learning-based dialogue systems. In Proceedings of the IEEE/ACL 2006 Workshop on Spoken Language Technology, pages 190–193, Aruba. find
  27. Rieser, Verena and Oliver Lemon. 2006c. Using machine learning to explore human multimodal clarification strategies. In Proceedings of the Poster Session of the 44th Meeting of the Association for Computational Linguistics, pages 659–666, Sydney. find
  28. Roy, Nicholas, Joelle Pineau, and Sebastian Thrun. 2000. Spoken dialog management for robots. In Proceedings of the 38th Meeting of the Association for Computational Linguistics, pages 93–100, Hong Kong. find
  29. Schatzmann, Jost, Kallirroi Georgila, and Steve Young. 2005. Quantitative evaluation of user simulation techniques for spoken dialogue systems. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 45–54, Lisbon. find
  30. Schatzmann, Jost, Matthew N. Stuttle, Karl Weilhammer, and Steve Young. 2005. Effects of the user model on simulation-based learning of dialogue strategies. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pages 220–225, San Juan, Puerto Rico. find
  31. Schatzmann, Jost, Karl Weilhammer, Matthew N. Stuttle, and Steve Young. 2006. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The Knowledge Engineering Review, 21:97–126. find
  32. Scheffler, Konrad and Steve Young. 2001. Corpus-based dialogue simulation for automatic strategy learning and evaluation. In Proceedings of the NAACL Workshop on Adaptation in Dialogue Systems, pages 64–70, Pittsburgh, PA. find
  33. Scheffler, Konrad and Steve Young. 2002. Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. In Proceedings of the Human Language Technology Conference, pages 12–19, San Diego, CA. find
  34. Shawe-Taylor, John and Nello Cristianini. 2004. Kernel Methods for Pattern Analysis. Cambridge University Press. find
  35. Singh, Satinder, Michael Kearns, Diane Litman, and Marilyn Walker. 2000a. Empirical evaluation of a reinforcement learning dialogue system. In Proceedings of the AAAI, pages 645–651, Whistler. find
  36. Singh, Satinder, Michael Kearns, Diane Litman, and Marilyn Walker. 2000b. Reinforcement learning for spoken dialogue systems. In Advances in Neural Information Processing Systems, 12:956–962. find
  37. Singh, Satinder, Diane Litman, Michael Kearns, and Marilyn Walker. 2002. Optimizing dialogue management with reinforcement learning: Experiments with the NJFun system. Journal of Artificial Intelligence Research (JAIR), 16:105–133. find
  38. Sutton, Richard and Andrew Barto. 1998. Reinforcement Learning. MIT Press, Cambridge, MA. find
  39. Walker, M., J. Aberdeen, J. Boland, E. Bratt, J. Garofolo, L. Hirschman, A. Le, S. Lee, S. Narayanan, K. Papineni, B. Pellom, B. Polifroni, A. Potamianos, P. Prabhu, A. Rudnicky, G. Sanders, S. Seneff, D. Stallard, and S. Whittaker. 2001. DARPA communicator dialog travel planning systems: The June 2… find
  40. Walker, M. and R. Passonneau. 2001. DATE: A dialogue act tagging scheme for evaluation of spoken dialogue systems. In Proceedings of the Human Language Technology Conference, pages 1–8, San Diego, CA.
  41. Walker, M., A. Rudnicky, J. Aberdeen, E. Bratt, J. Garofolo, H. Hastie, A. Le, B. Pellom, A. Potamianos, R. Passonneau, R. Prasad, S. Roukos, G. Sanders, S. Seneff, D. Stallard, and S. Whittaker. 2002a. DARPA Communicator Evaluation: Progress from 2000 to 2001. In Proceedings of the 7th Internationa… find
  42. Walker, M., A. Rudnicky, R. Prasad, J. Aberdeen, E. Bratt, J. Garofolo, H. Hastie, A. Le, B. Pellom, A. Potamianos, R. Passonneau, S. Roukos, G. Sanders, S. Seneff, and D. Stallard. 2002b. DARPA Communicator: Cross-system results for the 2001 evaluation. In Proceedings of the 7th International Confe… find
  43. Walker, Marilyn A., Jeanne C. Fromer, and Shrikanth Narayanan. 1998. Learning optimal dialogue strategies: A case study of a spoken dialogue agent for email. In Proceedings of the 17th International Conference on Computational Linguistics, pages 1345–1351, Montreal. find
  44. Walker, Marilyn A., Candace A. Kamm, and Diane J. Litman. 2000. Towards developing general models of usability with PARADISE. Natural Language Engineering, 6(3):363–377. find
  45. Walker, Marilyn A., Rebecca J. Passonneau, and Julie E. Boland. 2001. Quantitative and qualitative evaluation of DARPA Communicator spoken dialogue systems. In Proceedings of the 39th Meeting of the Association for Computational Linguistics, pages 515–522, Toulouse. find
  46. Williams, Jason. 2007. A method for evaluating and comparing user simulations: The Cramer-von Mises divergence. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pages 508–513, Kyoto. find
  47. Williams, Jason, Pascal Poupart, and Steve Young. 2005a. Factored partially observable Markov decision processes for dialogue management. In Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems, International Joint Conference on Artificial Intelligence (IJCAI), page… find
  48. Williams, Jason, Pascal Poupart, and Steve Young. 2005b. Partially observable Markov decision processes with continuous observations for dialogue management. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 25–34, Lisbon. find
  49. Williams, Jason and Steve Young. 2003. Using Wizard-of-Oz simulations to bootstrap reinforcement-learning-based dialog management systems. In Proceedings of the 4th SIGdial Workshop on Discourse and Dialogue, pages 135–139, Sapporo. find
  50. Williams, Jason and Steve Young. 2005. Scaling up POMDPs for dialog management: The “Summary POMDP” method. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pages 177–182, San Juan, Puerto Rico. find
  51. Young, Steve. 2000. Probabilistic methods in spoken dialogue systems. Philosophical Transactions of the Royal Society (Series A), 358(1769):1389–1402. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/henderson-2008-hybrid-reinforcement-supervised-learning-dialogue/