Elamur Research⌕
☰
home / papers / 2025 / τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment

τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment

Paper · 2025 · Victor Barres, Honghua Dong, Soham Ray et al.

Summary

Benchmarks for conversational agents, including the original τ-bench, usually let only the AI agent call tools, while the simulated user just supplies goals and facts. Real technical support is different: customers often have to act themselves, for example by rebooting a phone or switching settings. τ²-bench fills this gap with a "dual-control" setting, formalized as a Dec-POMDP, in which agent and user each have their own tools over a shared, changing world. The authors add a new Telecom troubleshooting domain and a compositional generator that builds verifiable tasks from small atomic scenarios, each defined by initialization, solution, and assertion functions. They also design a user simulator whose behavior is restricted by its tools and the environment state rather than by long prompts. Leading LLMs score well below perfect on Telecom tasks, and performance falls sharply when the agent must guide a user instead of acting alone. This separates reasoning failures from coordination and communication failures and identifies user guidance as a central weakness. The resource is released on GitHub (sierra-research/tau2-bench).

Key points

  • Passˆ1 on new Telecom tasks is about 34% for gpt-4.1, 42% for o4-mini, and 49% for claude-3.7-sonnet.
  • An ablation compares a "no-user" mode, where the agent controls every tool, with dual-control mode. Success drops by roughly 20 points of passˆ1 when the user must carry out the actions.
  • In manual annotation of gpt-4.1 traces, the Telecom user simulator made errors in 16% of conversations, with 6% critical. By comparison, retail had 40% (12% critical) and airline 47% (13% critical).
  • User persona matters: tasks with no persona specified often score on par with, or below, tasks using a "Hard" persona.
  • The authors list three limitations: domain building still depends on human experts, the tool-constrained simulator approach has not yet been applied to airline and retail, and the expert-novice knowledge gap is not modeled.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 654 works

Cite

Cite as Victor Barres, Honghua Dong, Soham Ray, Xujie Si and Karthik Narasimhan. 2025. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv:2506.07982.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.07982 · arXiv 2506.07982

@misc{barres2025bench,
  title = {{τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment}},
  author = {Victor Barres, Honghua Dong, Soham Ray, Xujie Si and Karthik Narasimhan},
  year = {2025},
  doi = {10.48550/arxiv.2506.07982},
  eprint = {2506.07982},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.07982}
}

References

27 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, et al. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556–571, 2020.
  2. Anthropic. Claude 3.7 Sonnet, 2025. Model release: 2025-02-24. find
  3. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278, 2018.
  4. Derek Chen, Howard Chen, Yi Yang, Alex Lin, and Zhou Yu. Action-based conversations dataset: A corpus for building more in-depth task-oriented dialogue systems. arXiv preprint arXiv:2104.00783, 2021. find
  5. Izzeddin Gür, Dilek Hakkani-Tür, Gokhan Tür, and Pararth Shah. User modeling for task oriented dialogues. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 900– 906, 2018. find
  6. He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling strategy and generation in negotiation dialogues. arXiv preprint arXiv:1808.09637, 2018.
  7. Zhiyuan Hu, Yue Feng, Anh Tuan Luu, Bryan Hooi, and Aldo Lipani. Unlocking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue system. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23. ACM, Oc… find
  8. Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128, 2023.
  9. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
  10. Taaha Kazi, Ruiliang Lyu, Sizhe Zhou, Dilek Hakkani-Tur, and Gokhan Tur. Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems, November 2024. arXiv:2411.09972. — [arXiv:2411.09972](https://arxiv.org/abs/2411.09972) find
  11. Elad Levi and Ilan Kadar. Intellagent: A multi-agent framework for evaluating conversational ai systems. arXiv preprint arXiv:2501.11067, 2025. — [arXiv:2501.11067](https://arxiv.org/abs/2501.11067) find
  12. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023.
  13. Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. arXiv preprint arXiv:2408.04682, 2024.
  14. Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016. find
  15. OpenAI. gpt-4.1, 2025. Model release: 2025-04-14. find
  16. OpenAI. o4-mini, 2025. Model release: 2025-04-16. find
  17. Olivier Pietquin and Helen Hastie. A survey on metrics for the evaluation of user simulations. The knowledge engineering review, 28(1):59–73, 2013. Publisher: Cambridge University Press. find
  18. Akshara Prabhakar, Zuxin Liu, Weiran Yao, Jianguo Zhang, Ming Zhu, Shiyu Wang, Zhiwei Liu, Tulika Awalgaonkar, Haolin Chen, Thai Hoang, et al. Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay. arXiv preprint arXiv:2504.03601, 2025.
  19. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. In ICLR, 2024.
  20. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. Identifying the risks of lm agents with an lm-emulated sandbox. arXiv preprint arXiv:2309.15817, 2023.
  21. Jost Schatzmann, Daniel Jurafsky, Michael Galley, and David Trevillian. Evaluating agenda-based user simulation for reinforcement learning of dialogue management. In Speech Communication, volume 47, pages 95–121, 2007. find
  22. Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. Flowbench: Revisiting and benchmarking workflow-guided planning for llm-based agents. arXiv preprint arXiv:2406.14884, 2024. find
  23. Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024. ↗
  24. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022.
  25. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024.
  26. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv preprint arXiv:2307.13854, 2023.
  27. Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, et al. Multiagentbench: Evaluating the collaboration and competition of llm agents. arXiv preprint arXiv:2503.01935, 2025. Broader Impact find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/barres-2025-tau-2-bench-evaluating-conversational/