Elamur Research⌕
☰
home / papers / 2026 / Efficient Agent Evaluation via Diversity-Guided User Simulation

Efficient Agent Evaluation via Diversity-Guided User Simulation

Paper · 2026 · Itay Nakash, George Kour and Ateret Anaby Tavor

Summary

Evaluating customer-facing LLM agents is costly and often incomplete. The usual approach repeats full conversations between the agent and a simulated user from scratch. This wastes tokens on nearly identical openings and rarely surfaces failures that depend on unusual user behaviour.

The authors (IBM Research) propose DIVERT, a user-simulation framework that treats conversations as trees rather than independent linear runs. After initial rollouts, it saves snapshots of the full agent, environment and dialogue state. An LLM-based junction chooser then picks a pivotal user turn. At that turn, a directed generator writes several alternative user replies that keep the original intent, and the reply least similar to the original (by embedding cosine similarity) is chosen. Execution resumes from the snapshot, so the shared prefix is reused, and new branches can be expanded again.

On multi-domain customer-service tasks, swapping some full rollouts for branches under the same trajectory budget finds more errors per token and exposes failures in more tasks. This makes agent reliability testing cheaper and more thorough.

Key points

  • Four-stage pipeline:
    • initial rollout plus state snapshotting;
    • junction selection;
    • diversity-guided user response generation (k = 3 candidates, scored with a sentence-transformers model);
    • resumption from the snapshot.
  • Ablation results (GPT-OSS-120B, 12-trajectory budget, 8 rollouts + 4 branches):
    • Errors per 100K tokens rose from 13.6 for full rollouts to 15.1 with the junction chooser, 15.8 with directed generation added, and 16.2 with diverse selection added.
    • Task failure count went from 78 to 75, then 80, then 81.
    • The junction chooser alone mainly saves cost. Directed, diverse user replies are needed to find failures in new tasks.
  • Diversity check:
    • The selected candidate is clearly further from the original reply than the other candidates (mean similarity 0.711 vs. 0.769 and 0.817, over 4,500 candidates).
    • The selected candidate also yields more divergent continuations (0.743 vs. 0.799 and 0.849, over 1,200 continuations).
  • Branches vs. rollouts: Adding rollouts alone soon stops uncovering new failing tasks, while increasing the number of branches keeps revealing them.
  • Compatibility with user policies: DIVERT works with any user policy, whether benign, adversarial or red-teaming. Future work includes branching on tool outputs and environment dynamics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States139 works · 70,691 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSoftware · 92 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Itay Nakash, George Kour and Ateret Anaby Tavor. 2026. Efficient Agent Evaluation via Diversity-Guided User Simulation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.112

@inproceedings{nakash2026efficient,
  title = {{Efficient Agent Evaluation via Diversity-Guided User Simulation}},
  author = {Itay Nakash, George Kour and Ateret Anaby Tavor},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.112},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.112}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.
  2. Elron Bandel, Asaf Yehudai, Lilach Eden, Yehoshua Sagron, Yotam Perlitz, Elad Venezian, Natalia Razinkov, Natan Ergas, Shlomit Shachor Ifergan, Segev Shlomov, Michal Jacovi, Leshem Choshen, Liat Ein-Dor, Yoav Katz, and Michal Shmueli-Scheuer. 2026. General agent evaluation. Preprint, arXiv:2602.2295… — [arXiv:2602.22953](https://arxiv.org/abs/2602.22953) find
  3. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. \\tauˆ2-bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982. find
  4. Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. Advances in Neural Information Processing Systems, 37:82895–82920. find
  5. Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, and James Zou. 2025. Cost-of-pass: An economic framework for evaluating language models. arXiv preprint arXiv:2504.13359. — [arXiv:2504.13359](https://arxiv.org/abs/2504.13359) find
  6. Google DeepMind. 2025. Gemini 2.5: Google’s latest multimodal model. https://deepmind.google/technologies/gemini/. ↗
  7. Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, and Vipin Chaudhary. 2025. Don’t pass@ k: A bayesian framework for large language model evaluation. arXiv preprint arXiv:2510.04265. — [arXiv:2510.04265](https://arxiv.org/abs/2510.04265) find
  8. Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. 2024. Ai agents that matter. arXiv preprint arXiv:2407.01502. find
  9. George Kour, Naama Zwerdling, Marcel Zalmanovici, Ateret Anaby Tavor, Ora Nova Fandina, and Eitan Farchi. 2025. Exploring straightforward methods for automatic conversational red-teaming. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computationa… find
  10. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
  11. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688.
  12. Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. 2025. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460.
  13. Tie Ma, Yixi Chen, Vaastav Anand, Alessandro Cornacchia, Amândio R Faustino, Guanheng Liu, Shan Zhang, Hongbin Luo, Suhaib A Fahmy, Zafar A Qazi, et al. 2026. Maestro: Multi-agent evaluation suite for testing, reliability, and observability. arXiv preprint arXiv:2601.00481. — [arXiv:2601.00481](https://arxiv.org/abs/2601.00481) find
  14. Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations.
  15. Itay Nakash, George Kour, Koren Lazar, Matan Vetzler, Guy Uziel, and Ateret Anaby Tavor. 2025a. Effective red-teaming of policy-adherent agents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2250–2268, Suzhou, China. Association for Computational Li… find
  16. Itay Nakash, George Kour, Guy Uziel, and Ateret Anaby Tavor. 2025b. Breaking ReAct agents: Foot-in-the-door attack will get you in. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6499–6524, Albuquerque, New Mexico. Association for Computational Linguistics. find
  17. Ivan Sekulić, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, Andre Ferreira Manso, and Roland Mathis. 2024. Reliable llm-based user simulator for task-oriented dialogue systems. arXiv preprint arXiv:2402.13374. find
  18. Jeonghoon Shim, Woojung Song, Cheyon Jin, Seung-won KooK, and Yohan Jo. 2025. Non-collaborative user simulators for tool agents. arXiv preprint arXiv:2509.23124. — [arXiv:2509.23124](https://arxiv.org/abs/2509.23124) find
  19. Sizhe Tang, Rongqian Chen, and Tian Lan. 2026. Agent alpha: Tree search unifying generation, exploration and evaluation for computer-use agents. arXiv preprint arXiv:2602.02995. — [arXiv:2602.02995](https://arxiv.org/abs/2602.02995) find
  20. Ningning Wang, Xavier Hu, Pai Liu, He Zhu, Yue Hou, Heyuan Huang, Shengyu Zhang, Jian Yang, Jiaheng Liu, Ge Zhang, et al. 2025. Efficient agents: Building effective agents while reducing cost. arXiv preprint arXiv:2508.02694. — [arXiv:2508.02694](https://arxiv.org/abs/2508.02694) find
  21. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045.
  22. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL https://arxiv.org/abs/2305.10601, 3:1. ↗
  23. Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2025. Survey on evaluation of llm-based agents. arXiv preprint arXiv:2503.16416. — [arXiv:2503.16416](https://arxiv.org/abs/2503.16416) find
  24. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023a. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406. find
  25. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023b. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nakash-2026-efficient-agent-evaluation-diversity-guided/