Elamur Research⌕
☰
home / papers / 2026 / DyBBT: Dynamic Balance via Bandit-inspired Targeting for Dialog Policy with Cognitive Dual Systems

DyBBT: Dynamic Balance via Bandit-inspired Targeting for Dialog Policy with Cognitive Dual Systems

Paper · 2026 · Shuyu Zhang, Yifan Wei, Jialuo Yuan et al.

Summary

Deep RL dialog policies for task-oriented systems usually explore with fixed rules such as ϵ-greedy. These rules ignore how the value of exploring shifts over the course of a conversation. Evolutionary alternatives like EIERL scale poorly, and LLM deliberation is too costly to run at every turn. DyBBT treats the choice of reasoning mode as a contextual multi-armed bandit. Each turn is compressed into a three-dimensional cognitive state that encodes how far the dialog has advanced, how uncertain the user's intent is, and how strongly slots constrain one another. A lightweight meta-controller uses this state to pick one of two systems:

  • System 1: a fast LoRA-tuned LLM policy trained with SFT and PPO.
  • System 2: a frozen backbone LLM that performs multi-path deliberation.

System 2 is invoked only for rarely visited states or when System 1 lacks confidence. Its outputs are then distilled back into System 1. Assuming Lipschitz-smooth rewards, the authors argue for sublinear regret. On MS Dialog and MultiWOZ, DyBBT reports top results in task success, efficiency and generalization, and human judges find its choices consistent with expert decisions. The approach shows how expensive reasoning can be spent selectively.

Key points

  • The cognitive state c_t = [d_t, u_t, ρ_t] covers dialog stage, user uncertainty and slot dependency degree. The bandit arms are System 1 and System 2.
  • System 2 is triggered by a UCB-inspired count condition, n_t(c_t) < τ·log T, combined with a check of System 1's confidence against a threshold κ.
  • Empirical cumulative regret grows roughly like √T, which the authors read as indirect support for their Lipschitz and approximate-MDP assumptions.
  • The failure analysis names three causes: limits of the handcrafted cognitive state, errors in System 2 that propagate through distillation, and coarse binning of C. These failures affect about 5.2% of dialogs.
  • The evaluation uses MS Dialog and MultiWOZ with simulated users, plus human evaluation and real-user studies. Code is available at github.com/carsonz/DyBBT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,053 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🏛️Chinese Academy of SciencesBeijing · China246 works · 24,951 citations🎓The University of SydneySydney · Australia49 works · 68,350 citations🎓Beihang UniversityBeijing · China86 works · 9,097 citations🎓Beijing Institute of Graphic CommunicationBeijing · China3 works

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksDecision Sciences · 249 worksManagement Science and Operations Research · 134 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 654 works

Cite

Cite as Shuyu Zhang, Yifan Wei, Jialuo Yuan, Xinru Wang, Yanmin Zhu, Yujie Liu and Bin Li. 2026. DyBBT: Dynamic Balance via Bandit-inspired Targeting for Dialog Policy with Cognitive Dual Systems. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2180

@inproceedings{zhang2026dybbt,
  title = {{DyBBT: Dynamic Balance via Bandit-inspired Targeting for Dialog Policy with Cognitive Dual Systems}},
  author = {Shuyu Zhang, Yifan Wei, Jialuo Yuan, Xinru Wang, Yanmin Zhu, Yujie Liu and Bin Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2180},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2180}
}

References

49 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kavosh Asadi, Dipendra Misra, and Michael L. Littman. 2018. Lipschitz continuity in model-based reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, ICML’18. find
  2. Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. 2011. Pure exploration in finitely-armed and continuous-armed bandits. Theor. Comput. Sci., 412(19):1832–1852. find
  3. Tao He, Lizi Liao, Yixin Cao, Yuanxing Liu, Ming Liu, Zerui Chen, and Bing Qin. 2024. Planning like human: A dual-process framework for dialogue planning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4768–4791, Bangkok, Tha… find
  4. Xiuyi Chen, Fandong Meng, Peng Li, Feilong Chen, Shuang Xu, Bo Xu, and Jie Zhou. 2020. Bridging the gap between prior and posterior knowledge selection for knowledge-grounded dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages… find
  5. Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang, Min Yang, Fei Huang, Luo Si, Jian Sun, and Yongbin Li. 2022. Galaxy: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection. Proceedings of the AAAI Confere… find
  6. Richard Dearden, Nir Friedman, and Stuart Russell. 1998. Bayesian q-learning. In Proceedings of the Fifteenth National Conference on Artificial Intelligence and Tenth Innovative Applications of Artificial Intelligence Conference, AAAI 98, IAAI 98, July 26-30, 1998, Madison, Wisconsin, USA, pages 761… find
  7. Xu Jia, Ruochen Zhang, and Min Peng. 2024. Multi-domain gate and interactive dual attention for multi-domain dialogue state tracking. Knowledge-Based Systems, 286:111383. find
  8. Wenjie Dong, Sirong Chen, and Yan Yang. 2025. Pro-TOD: Proactive task-oriented dialogue system based on large language model. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9147–9164, Abu Dhabi, UAE. Association for Computational Linguistics. find
  9. Yihan Du, R. Srikant, and Wei Chen. 2024. Cascading reinforcement learning. In International Conference on Representation Learning, volume 2024, pages 30263–30304. find
  10. Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. MultiWOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In Proceedings of the Twelfth Language… find
  11. Dylan J. Foster and Alexander Rakhlin. 2020. Beyond ucb: optimal and efficient contextual bandits with regression oracles. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org. find
  12. Aurélien Garivier and Eric Moulines. 2011. On upper-confidence bound policies for switching bandit problems. In Algorithmic Learning Theory, pages 174– 188, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  13. James Jerome Gibson. 1979. The Ecological Approach to Visual Perception. Houghton Mifflin, Boston. Original edition. find
  14. Shuai Han, Wenbo Zhou, Jiayi Lu, Jing Liu, and Shuai Lü. 2022. Nrowan-dqn: A stable noisy network with noise reduction and online weight adjustment for exploration. Expert Systems with Applications, 203:117343. find
  15. Saurav Kadavath, Tom Conerly, and et al. 2022. Language models (mostly) know what they know. Preprint, arXiv:2207.05221. find
  16. Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. 2008. Multi-armed bandits in metric spaces. In Proceedings of the Fortieth Annual ACM Symposium on Theory of Computing, STOC ’08, page 681–690, New York, NY, USA. Association for Computing Machinery. find
  17. Junpei Komiyama, Edouard Fouché, and Junya Honda. 2024. Finite-time analysis of globally nonstationary multi-armed bandits. Journal of Machine Learning Research, 25(112):1–56. find
  18. Kahneman, D. (2011): Thinking, fast and slow. Statistical Papers, 55(3):915–915. find
  19. Wai-Chung Kwan, Hong-Ru Wang, Hui-Min Wang, and Kam-Fai Wong. 2023. A survey on recent advances and challenges in reinforcement learning methods for task-oriented dialogue policy learning. Machine Intelligence Research, 20:1–17. find
  20. Jing Yang Lee, Kong Aik Lee, and Woon Seng Gan. 2023. An empirical Bayes framework for open-domain dialogue generation. In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 192–204, Singapore. Association for Computational Linguistics. find
  21. Changqun Li, Linlin Wang, Xin Lin, Shizhou Huang, and Liang He. 2024. Hypernetwork-assisted parameter-efficient fine-tuning with meta-knowledge distillation for domain knowledge disentanglement. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1681–1695, Mexico City, M… find
  22. Xiujun Li, Yu Wang, Siqi Sun, Sarah Panda, Jingjing Liu, and Jianfeng Gao. 2018. Microsoft dialogue challenge: Building end-to-end task-completion dialogue systems. Preprint, arXiv:1807.11125. — [arXiv:1807.11125](https://arxiv.org/abs/1807.11125) find
  23. Anthony Liang, Guy Tennenholtz, Chih-Wei Hsu, Yinlam Chow, Erdem Biyik, and Craig Boutilier. 2024. Dynamite-rl: a dynamic model for improved temporal meta-reinforcement learning. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA… find
  24. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research. find
  25. Yuanguo Lin, Fan Lin, Guorong Cai, Hong Chen, Linxin Zou, Yunxuan Liu, and Pengcheng Wu. 2025. Evolutionary reinforcement learning: A systematic review and future directions. Mathematics, 13(5). find
  26. Shuai Ma, Qiaoyi Chen, Xinru Wang, Chengbo Zheng, Zhenhui Peng, Ming Yin, and Xiaojuan Ma. 2025. Towards human-ai deliberation: Design and evaluation of llm-empowered deliberative ai for ai-assisted decision-making. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI… find
  27. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, and et al. 2015. Human-level control through deep reinforcement learning. nature, 518(7540):529–533. find
  28. Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. 2017. Bridging the gap between value and policy based reinforcement learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pages 2772–2782, Red Hook, NY, USA. Curran Associates In… find
  29. Xuecheng Niu, Akinori Ito, and Takashi Nose. 2024. Scheduled curiosity-deep dyna-q: Efficient exploration for dialog policy learning. IEEE Access, 12:46940–46952. find
  30. Ronald Ortner and Daniil Ryabko. 2012. Online regret bounds for undiscounted continuous reinforcement learning. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’12, page 1763–1771, Red Hook, NY, USA. Curran Associates Inc. find
  31. Jason Pazis and Ronald Parr. 2013. Pac optimal exploration in continuous space markov decision processes. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, AAAI’13, page 774–781. AAAI Press. find
  32. Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. 2018. Deep Dyna-Q: Integrating planning for task-completion dialogue policy learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2182–2192, Melbourne,… find
  33. Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. 2017. find
  34. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2231– 2240, Copenhagen, Denmark. Association for Computational Linguistics. find
  35. Libo Qin, Wenbo Pan, Qiguang Chen, Lizi Liao, Zhou Yu, Yue Zhang, Wanxiang Che, and Min Li. 2023. End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5925–5941, Singapor… find
  36. Mahdin Rohmatillah and Jen-Tzung Chien. 2023. Hierarchical reinforcement learning with guidance for multi-domain dialogue policy. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:748–761. find
  37. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. find
  38. Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. 2024. Math-LLaVA: Bootstrapping mathematical reasoning for multimodal large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4663–4680, Miami, Florid…
  39. Olivier Sigaud. 2023. Combining evolution and deep reinforcement learning for policy search: A survey. ACM Trans. Evol. Learn. Optim., 3(3). find
  40. David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML’14, page I–387–I–395. JMLR.org. find
  41. Jingtao Sun, Jiayin Kou, Wenyan Hou, and Yujei Bai. 2025. A multi-agent curiosity reward model for task-oriented dialogue systems. Pattern Recognition, 157:110884. find
  42. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computation…
  43. Heng-Da Xu, Xian-Ling Mao, Puhai Yang, Fanshu Sun, and Heyan Huang. 2024. Rethinking task-oriented dialogue systems: From complex modularity to zero-shot autonomous agent. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2748–… find
  44. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, volume 36, pages 11809–11822. Curran Associates, Inc. find
  45. Zihao Yi, Jiarui Ouyang, Zhe Xu, Yuwen Liu, Tianhao Liao, Haohao Luo, and Ying Shen. 2025. A survey on recent advances in llm-based multi-turn dialogue systems. ACM Comput. Surv., 58(6).
  46. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don’t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653–8665, Toronto, Canada. Association for Computational Linguistics. find
  47. Jiahao Ying, Yixin Cao, Kai Xiong, Long Cui, Yidong He, and Yongbin Liu. 2024. Intuitive or dependent? investigating LLMs’ behavior style to conflicting prompts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4221–4246, Bangk… find
  48. Jianxing Yu, Shiqi Wang, Han Yin, Qi Chen, Wei Liu, Yanghui Rao, and Qinliang Su. 2025. Diversified generation of commonsense reasoning questions. Expert Systems with Applications, 263:125776. find
  49. Ming Zhang, Caishuang Huang, Yilong Wu, Shichun Liu, Huiyuan Zheng, Yurui Dong, Yujiong Shen, Shihan Dou, Jun Zhao, Junjie Ye, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. TransferTOD: A generalizable Chinese multi-domain task-oriented dialogue system with transfer capabilities. In Proceedings of th… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-dybbt-dynamic-balance-bandit-inspired/