Elamur Research⌕
☰
home / papers / 2023 / Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents

Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents

Paper · 2023 · Yang Deng, Wenxuan Zhang, Wai Lam et al.

Summary

LLM-based chat assistants tend to follow the user's lead. This makes them weak at proactive dialogues such as negotiation, emotional support and tutoring, where the agent has to steer the conversation toward a goal. Earlier fixes either prompt a frozen LLM to plan its strategy, which is capped by that model's own planning skill, or refine a plan case by case through self-play feedback, which does not carry over to new cases. The authors propose PPDPP, a small tunable language model that plugs into a frozen LLM and predicts which strategy the agent should use next. Training has two stages. First, supervised fine-tuning on human-annotated dialogues. Second, reinforcement learning on conversations simulated between two LLMs (assistant and user), where a third LLM gives verbal judgements of goal progress that are converted into scalar rewards. The paper also introduces an interactive evaluation that measures success rate and average turns to reach the goal. Across all three tasks, PPDPP consistently beats prompt-based baselines. Because the planner is a swappable module, it generalises to new cases and moves between applications without retraining the LLM.

Key points

  • The dialogue is framed as a Markov Decision Process. At each turn, the plug-in picks an action from a set of strategies predefined by domain experts, and the policy is optimised for cumulative long-term reward.
  • Goal-oriented AI feedback turns an LLM's textual assessment of goal achievement into scalar rewards. These rewards are based on whole multi-turn episodes rather than preferences over single responses.
  • Baselines compared include MI-Prompt, Ask-an-Expert, ProCoT and ICL-AIF. Table 1 contrasts these methods with PPDPP along seven properties, including transferability and gradient-based learning.
  • In one emotional-support example, PPDPP reached the goal in 3 turns, compared with 4 for ProCoT, 5 for ICL-AIF and 7 for Ask-an-Expert. The aggregate scores are in a part of the excerpt that was omitted.
  • Code is available at github.com/dengyang17/PPDPP.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore221 works · 39,953 citations🎓Chinese University of Hong KongHong Kong · China176 works · 17,536 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 664 works

Cite

Cite as Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng and Tat-Seng Chua. 2023. Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents. arXiv:2311.00262.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.00262 · arXiv 2311.00262

@misc{deng2023plug,
  title = {{Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents}},
  author = {Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng and Tat-Seng Chua},
  year = {2023},
  doi = {10.48550/arxiv.2311.00262},
  eprint = {2311.00262},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.00262}
}

References

53 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — Constitutional AI: Harmlessness from AI Feedback
  2. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. CoRR, abs/2302.04023, 20…
  3. Maximillian Chen, Xiao Yu, Weiyan Shi, Urvi Awasthi, and Zhou Yu. Controllable mixed-initiative dialogue generation through prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023, pp. 951–966, 2023. URL https://aclanthology.org/2023.acl-short.… ↗
  4. Yi Cheng, Wenge Liu, Wenjie Li, Jiashuo Wang, Ruihui Zhao, Bang Liu, Xiaodan Liang, and Yefeng Zheng. Improving multi-turn emotional support dialogue generation with lookahead strategy planning. CoRR, abs/2210.04242, 2022. URL https://doi.org/10.48550/arXiv. 2210.04242. ↗
  5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/. ↗
  6. Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. Unified conversational recommendation policy learning via graph-based reinforcement learning. In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1431–1441. ACM, 2021. URL https:/… — [doi:10.1145/3404835.3462913](https://doi.org/10.1145/3404835.3462913) find
  7. Yang Deng, Wenqiang Lei, Wenxuan Zhang, Wai Lam, and Tat-Seng Chua. PACIFIC: towards proactive conversational question answering over tabular and textual data in finance. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pp. 6970–6984, 2022. URL h… find
  8. Yang Deng, Wenqiang Lei, Wai Lam, and Tat-Seng Chua. A survey on proactive dialogue systems: Problems, methods, and prospects. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, pp. 6583–6591, 2023a. URL https://doi.org/10.24963/ijcai.2023/738. — [doi:10.24963/ijcai.2023/738](https://doi.org/10.24963/ijcai.2023/738) ↗
  9. Yang Deng, Wenqiang Lei, Lizi Liao, and Tat-Seng Chua. Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration. CoRR, abs/2305.13626, 2023b. URL https://doi.org/10.48550/arXiv.2305.13626. ↗
  10. Yang Deng, Wenxuan Zhang, Yifei Yuan, and Wai Lam. Knowledge-enhanced mixed-initiative dialogue system for emotional support conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, pp. 4079– 4095, 2023c. URL https:/… find
  11. Yihao Feng, Shentao Yang, Shujian Zhang, Jianguo Zhang, Caiming Xiong, Mingyuan Zhou, and Huan Wang. Fantastic rewards and how to tame them: A case study on reward learning for task-oriented dialogue systems. In The Eleventh International Conference on Learning Representations, ICLR 2023, 2023. URL… find
  12. Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from AI feedback. CoRR, abs/2305.10142, 2023. URL https://doi.org/10.48550/arXiv.2305.10142. ↗
  13. Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. Advances and challenges in conversational recommender systems: A survey. AI Open, 2:100–126, 2021. URL https://doi.org/10.1016/j.aiopen.2021.06.002. ↗
  14. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. CoRR, abs/2305.14992, 2023. URL https://doi.org/10.48550/arXiv.2305.14992. ↗
  15. He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2333–2343, 2018. URL https://doi.org/10.18653/v1/ d18-1256.
  16. Clara E Hill. Helping skills: Facilitating, exploration, insight, and action. 2009. find
  17. Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. Gpt-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems. In The Tenth International Conference on Learning Representations, ICLR 2022, 2022. URL https://openreview.net/forum?id= qaxhBG1UUaS. ↗
  18. Rishabh Joshi, Vidhisha Balachandran, Shikhar Vashishth, Alan W. Black, and Yulia Tsvetkov. Dialograph: Incorporating interpretable strategy-graph networks into negotiation dialogues. In 9th International Conference on Learning Representations, ICLR 2021, 2021. URL https://openreview.net/forum?id=k… ↗
  19. Wai-Chung Kwan, Hongru Wang, Huimin Wang, and Kam-Fai Wong. A survey on recent advances and challenges in reinforcement learning methods for task-oriented dialogue policy learning. Int. J. Autom. Comput., 20(3):318–334, 2023. URL https://doi.org/10.1007/ s11633-022-1347-y. find
  20. Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. Guiding large language models via directional stimulus prompting. CoRR, abs/2302.11520, 2023. URL https://doi.org/10.48550/arXiv.2302.11520. — [doi:10.48550/arxiv.2302.11520](https://doi.org/10.48550/arxiv.2302.11520) ↗
  21. Lizi Liao, Grace Hui Yang, and Chirag Shah. Proactive conversational agents. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, pp. 1244–1247, 2023. URL https://doi.org/10.1145/3539597.3572724. — [doi:10.1145/3539597.3572724](https://doi.org/10.1145/3539597.3572724) ↗
  22. Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language…
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. URL http://arxiv.org/abs/1907.11692. ↗
  24. Jakub Macina, Nico Daheim, Lingzhi Wang, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. Opportunities and challenges in neural dialog tutoring. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, pp. 2349–2364, 20…
  25. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. CoRR, abs… find
  26. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training… — Training language models to follow instructions with human feedback
  27. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. CoRR, abs/2302.04761, 2023. URL https://doi.org/10.48550/arXiv. 2302.04761. ↗
  28. Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. Minding language models’ (lack of) theory of mind: A plug-and-play multi-character belief tracker. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… — Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker
  29. Ivan Sekulic, Mohammad Aliannejadi, and Fabio Crestani. Evaluating mixed-initiative conversational search systems via user simulation. In WSDM ’22: The Fifteenth ACM International Conference on Web Search and Data Mining, pp. 888–896, 2022. find
  30. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving AI tasks with chatgpt and its friends in huggingface. CoRR, abs/2303.17580, 2023. URL https://doi.org/10.48550/arXiv.2303.17580. ↗
  31. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. CoRR, abs/2303.11366, 2023. URL https://doi.org/10. 48550/arXiv.2303.11366.
  32. Katherine Stasaski, Kimberly Kao, and Marti A. Hearst. CIMA: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, BEA@ACL 2020, pp. 52–64, 2020. URL https://doi.org/10.18653/v1/2020.bea-1.5. ↗
  33. Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12, pp. 1057–1063, 1999. find
  34. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  35. Hongru Wang, Rui Wang, Fei Mi, Yang Deng, Zezhong Wang, Bin Liang, Ruifeng Xu, and Kam-Fai Wong. Cue-cot: Chain-of-thought prompting for responding to in-depth dialogue questions with llms. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 12047– 12064, 2023a. URL https:/… find
  36. Hongru Wang, Rui Wang, Fei Mi, Yang Deng, Zezhong Wang, Bin Liang, Ruifeng Xu, and Kam-Fai Wong. Cue-cot: Chain-of-thought prompting for responding to in-depth dialogue questions with llms. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 12047– 12064. Association for Co… find
  37. Lingzhi Wang, Mrinmaya Sachan, Xingshan Zeng, and Kam-Fai Wong. Strategize before teaching: A conversational tutoring system with pedagogy self-distillation. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 2223–2229, 2023c. URL https://aclanthology.org/2023.findings-eac… ↗
  38. Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. Rethinking the evaluation for conversational recommendation in the era of large language models. CoRR, abs/2305.13112, 2023d. find
  39. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR 2023.
  40. OpenReview.net, 2023e. URL https://openreview.net/pdf? id=1PL1NIMMrw. ↗
  41. Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR, abs/2303.04671, 2023. URL https://doi.org/10.48550/arXiv.2303.04671. ↗
  42. Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, and Julian J. McAuley. Small models are valuable plug-ins for large language models. CoRR, abs/2305.08848, 2023. URL https://doi.org/10.48550/arXiv.2305.08848. ↗
  43. Runzhe Yang, Jingxiao Chen, and Karthik Narasimhan. Improving dialog systems for negotiation with personality modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 20… find
  44. Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. Retroformer: Retrospective large language agents with policy gradient optimization, 2023. find
  45. Xiao Yu, Maximillian Chen, and Zhou Yu. Prompt-based monte-carlo tree search for goal-oriented dialogue policy planning. CoRR, abs/2305.13660, 2023. URL https://doi.org/10. 48550/arXiv.2305.13660. find
  46. Haolan Zhan, Yufei Wang, Tao Feng, Yuncheng Hua, Suraj Sharma, Zhuang Li, Lizhen Qu, and Gholamreza Haffari. Let’s negotiate! A survey of negotiation dialogue systems. CoRR, 2022. find
  47. Qiang Zhang, Jason Naradowsky, and Yusuke Miyao. Ask an expert: Leveraging language models to improve strategic reasoning in goal-oriented dialogue models. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 6665–6694, 2023a. URL https://aclanthology.org/2023.findings-acl.41… ↗
  48. Xiaoying Zhang, Baolin Peng, Kun Li, Jingyan Zhou, and Helen Meng. Sgp-tod: Building task bots effortlessly via schema-guided llm prompting, 2023b. find
  49. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. DIALOGPT : Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistic… — DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
  50. Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang, Yanpeng Tong, and Bing Qin. Is chatgpt equipped with emotional dialogue capabilities? CoRR, abs/2304.09582, 2023. URL https://doi.org/10.48550/arXiv.2304.09582. — [doi:10.48550/arxiv.2304.09582](https://doi.org/10.48550/arxiv.2304.09582) ↗
  51. Zhonghua Zheng, Lizi Liao, Yang Deng, and Liqiang Nie. Building emotional support chatbots in the era of llms. CoRR, abs/2308.11584, 2023. URL https://doi.org/10.48550/ arXiv.2308.11584. — [arXiv:2308.11584](https://arxiv.org/abs/2308.11584) find
  52. Yiheng Zhou, He He, Alan W. Black, and Yulia Tsvetkov. A dynamic strategy coach for effective negotiation. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, SIGdial 2019, pp. 367–378, 2019. URL https://doi.org/10.18653/v1/W19-5943. ↗
  53. Yiheng Zhou, Yulia Tsvetkov, Alan W. Black, and Zhou Yu. Augmenting non-collaborative dialog systems with explicit semantic and strategic dialog history. In 8th International Conference on Learning Representations, ICLR 2020, 2020. URL https://openreview.net/forum? id=ryxQuANKPB. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/deng-2023-plug-play-policy-planner-large/