Elamur Research⌕
☰
home / papers / 2026 / When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue

When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue

Paper · 2026 · Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou et al.

Summary

Current evaluators of conversational agents typically check either user satisfaction or the correctness of tool calls in isolation. As a result, they overlook multi-turn failures where the agent mishandles a tool, for example by misreading its output or claiming a failed action succeeded, while the user remains content. To study this blind spot, the authors build TRACE, a synthetic English benchmark. Its conversations are generated by Claude Sonnet-4 and DeepSeek-R1 over 30 tools taken from existing datasets and labeled along four dimensions: tool correctness, agent behavior, user satisfaction, and overall success. Quality control combines a human-verified gold subset with an LLM-judge-filtered silver subset. The authors also introduce SCOPE, an extension of SPUR that discovers multiple evaluation areas, derives rubrics for each area, and weights criteria by severity, including make-or-break rules. SCOPE achieves the best overall F1, but every tested evaluator performs poorly on "hard negatives," where a satisfied user masks an agent error. This shows that user-satisfaction signals alone are not a sufficient basis for evaluating tool-using agents.

Key points

  • TRACE composition: 566 conversations (2,181 agent-user turns) spanning 26 manually designed situations.
    • Gold subset: 191 conversations (39 POS / 152 NEG).
    • Silver subset: 375 conversations (143 POS / 232 NEG).
  • Tools: 28 from ToolTalk, 1 from MINT (wikipedia_search), and 1 from API-Bank (calculator), grouped into nine tool groups. Tool execution is simulated by the LLM.
  • Quality control:
    • Human annotation reached 80.7% agreement (Cohen's Kappa 0.385).
    • The Sonnet-4 LLM judge used for silver filtering has 93.13% precision.
  • Main results (Sonnet-3.5): SCOPE reaches 0.79 accuracy and 0.76 F1, outperforming SPUR.
    • On hard negatives, SCOPE scores 0.32 (0.42 with GPT-4.1), compared with SPUR's near-zero performance.
    • The Skywork reward model performs best on hard negatives (0.70) but drops to 0.86 on easy cases (vs. SCOPE's 0.93), giving a slightly lower F1 of 0.75.
  • Ablations: Removing rubric weighting reduces SCOPE accuracy by 13.9% with Sonnet-3.5 and 37.97% with GPT-4.1. Removing make-or-break weights reduces it by 19% and 35.44%, respectively.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou, Shamik Roy, Vinayshekhar Bannihatti Kumar and Rashmi Gangadharaiah. 2026. When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.72

@inproceedings{shourya2026when,
  title = {{When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue}},
  author = {Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou, Shamik Roy, Vinayshekhar Bannihatti Kumar and Rashmi Gangadharaiah},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.72},
  url = {https://doi.org/10.18653/v1/2026.gem-main.72}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. \#\# Limitations Anthropic. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/ ↗
  2. We identify the following limitations of our work. in Natural Language Processing, pages 3102–3116, Stroudsburg, PA, USA. Association for Computational Linguistics. claude-3-5-sonnet. find
  3. Anthropic. 2025. Introducing claude 4. https://www.anthropic.com/news/claude-4. ↗
  4. Samuel Arcadinho, David Aparicio, and Mariana Almeida. 2024. Automated test generation to evaluate tool-augmented LLMs as conversational AI agents. arXiv [cs.CL]. find
  5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 81 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llm…
  6. Michelle Elizabeth, Alicja Kasicka, Natalia Krawczyk, Magalie Ochs, Gwénolé Lecorvé, Justyna Gromada, and Lina M Rojas Barahona. 2025. Neural models and language model prompting for the multidimensional evaluation of open-ended conversations. In Proceedings of the Twelfth Dialog System Technology Ch… find
  7. Nicholas Farn and Richard Shin. 2023. ToolTalk: Evaluating tool-usage in a conversational setting. arXiv [cs.CL]. find
  8. Shengyue Guan, Haoyi Xiong, Jindong Wang, Jiang Bian, Bin Zhu, and Jian-Guang Lou. 2025. Evaluating LLM-based agents for multi-turn conversations: A survey. arXiv [cs.CL]. find
  9. Zishan Guo, Yufei Huang, and Deyi Xiong. 2024. CToolEval: A chinese benchmark for LLM-powered agent evaluation in real-world API interactions. In Findings of the Association for Computational Linguistics ACL 2024, pages 15711–15724, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  10. Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruifeng Xu, and Qun Liu. 2024. Planning, creation, usage: Benchmarking LLMs for comprehensive tool utilization in real-world complex scenarios. In Findings of the… find
  11. Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2023. Meta-Tool benchmark for large language models: Deciding whether to use tools and which to use. In The Twelfth International Conference on Learning Representat… find
  12. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods Ying-Chun Lin, Jennifer Neville, Jack Stokes, Longqi Yang, Tara…
  13. Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. 2025. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352.
  14. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. In EMNLP.
  15. John Mendonça, Alon Lavie, and Isabel Trancoso. 2024. On the benchmarking of LLMs for open-domain dialogue evaluation. In Proceedings of the 6th Workshop on NLP for Conversational AI (NLP4ConvAI 2024), pages 1–12. find
  16. John Mendonça, Patrícia Pereira, Helena Moniz, Joao Paulo Carvalho, Alon Lavie, and Isabel Trancoso. 2023. Simple LLM prompting is state-of-the-art for robust and multilingual dialogue evaluation. In Proceedings of the Eleventh Dialog System Technology Challenge, pages 133–143. find
  17. OpenAI. 2025. Introducing gpt-4.1 in the api. https://openai.com/index/gpt-4-1/. Model announcement and technical overview. ↗
  18. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToolLLM: Facilitating large language models to master 16000+ real…
  19. Weiwei Sun, Shuo Zhang, Krisztian Balog, Zhaochun Ren, Pengjie Ren, Zhumin Chen, and Maarten de Rijke. 2021. Simulating user satisfaction for the evaluation of task-oriented dialogue systems. In Proceedings of the 44rd International ACM SIGIR Conference on Research and Development in Information Ret… find
  20. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. ToolAlpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv [cs.CL].
  21. Pei Wang, Yanan Wu, Noah Wang, Jiaheng Liu, Xiaoshuai Song, Z Y Peng, Ken Deng, Chenchen Zhang, JiakaiWang, Junran Peng, Ge Zhang, Hangyu Guo, Zhaoxiang Zhang, Wenbo Su, and Bo Zheng. 2024. MTU-bench: A multi-granularity tool-use benchmark for large language models. In The Thirteenth International C… find
  22. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. In The Twelfth International Conference on Learning Representations.
  23. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the tool manipulation capability of open-source large language models. arXiv [cs.CL].
  24. Rui Yu, Shenghua Wan, Yucen Wang, Chen-Xiao Gao, Le Gan, Zongzhang Zhang, and De-Chuan Zhan. 2025. Reward models in deep reinforcement learning: A survey. arXiv preprint arXiv:2506.15421. — [arXiv:2506.15421](https://arxiv.org/abs/2506.15421) find
  25. Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A dataset for LLM question answering with external tools. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shourya-2026-when-users-are-happy-but/