Elamur Research⌕
☰
home / papers / 2024 / StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models

StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models

Paper · 2024 · Zhicheng Guo, Sijie Cheng, Hao Wang et al.

Summary

Benchmarks for tool learning in LLMs face a trade-off. Small sets of curated tools are reliable but unrealistic. Large collections of real online APIs, as in ToolBench, are realistic but change over time, so scores cannot be reproduced. The authors re-ran ToolBench baselines on the I1-Instruction group and got consistently lower pass rates than originally reported. They traced the instability to two sources: many APIs had broken or changed, and the gpt-3.5-based judge was unreliable. Their answer is StableToolBench, which keeps ToolBench's tasks but adds two components. The first is a virtual API server: it checks a large response cache first, and when a call is neither cached nor answerable by the real API, a GPT-4 simulator generates a response from the API documentation and cached example calls. The second is a revised evaluation that first filters for solvable tasks and uses GPT-4 as the judge. Experiments show far more stable scores under API failures, and the simulated APIs are comparable to real ones in realism and diversity. This makes long-term model comparison on tool use more trustworthy.

Key points

  • A scan of ToolBench's APIs found that only 44.4% of calls succeeded. The main failures were parsing errors (25.9%, fixed in the new benchmark), unreachable endpoints (14.8%), missing authorisation (6.4%), changed parameters (3.6%) and "not found" errors (3.5%).
  • Cache entries are keyed by category, tool, API name and arguments. The cache was seeded from ToolBench training and test call histories, only valid records are kept, and new calls keep extending it.
  • Two new metrics, SoPR (Solvable Pass Rate) and SoWR (Solvable Win Rate), take the place of ToolBench's Pass Rate and Win Rate, where unsure or unsolvable judgements introduced randomness.
  • Three annotators manually checked the automatic evaluators (Claude 2, Gemini Pro, GPT-3.5 Turbo, GPT-4 Turbo) on solvability, answer-solving and comparison questions. This check motivated replacing gpt-3.5 as the judge.
  • Limitations: relying on GPT-4 for both simulation and evaluation is costly and exposes the benchmark to future model updates. The cache may also cover future methods less well. Code is available at THUNLP-MT/StableToolBench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States533 works · 443,885 citations🎓Tsinghua UniversityBeijing · China439 works · 94,783 citations🎓University of Hong KongPok Fu Lam · China127 works · 13,824 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

Cite

Cite as Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun and Yang Liu. 2024. StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models. arXiv:2403.07714.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.07714 · arXiv 2403.07714

@misc{guo2024stabletoolbench,
  title = {{StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models}},
  author = {Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun and Yang Liu},
  year = {2024},
  doi = {10.48550/arxiv.2403.07714},
  eprint = {2403.07714},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.07714}
}

References

38 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  2. Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2024. Large Language Models as Tool Makers. In Proc. of The Twelfth International Conference on Learning Representations (ICLR 2024).
  3. Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023a. FireAct: Toward language agent fine-tuning. ArXiv preprint, abs/2310.05915.
  4. Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. 2023b. T-Eval: Evaluating the Tool Utilization Capability Step by Step. ArXiv preprint, abs/2312.14033. — [arXiv:2312.14033](https://arxiv.org/abs/2312.14033) find
  5. Gemini Team. 2023. Gemini: A Family of Highly Capable Multimodal Models.
  6. Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual programming: Compositional visual reasoning without training. In In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), pages 14953–14962.
  7. Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023. ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings. ArXiv preprint, abs/2305.11554.
  8. Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. 2023. Tool documentation enables zero-shot tool-usage with large language models. ArXiv preprint, abs/2308.00675. find
  9. Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. 2023. MetaTool benchmark for large language models: Deciding whether to use tools and which to use. ArXiv preprint, abs/2310.03128.
  10. Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. 2024. GeneGPT: augmenting large language models with domain tools for improved access to biomedical information. Bioinformatics, 40(2):btae075.
  11. Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023a. API-Bank: A Benchmark for Tool-Augmented LLMs. find
  12. Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023b. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. ArXiv preprint, abs/2304.08244.
  13. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. ArXiv preprint, abs/2304.09842.
  14. Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018. UMAP: Uniform Manifold Approximation and Projection. The Journal of Open Source Software, 3(29):861. find
  15. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. 2023. Augmented Language Models: A Survey. ArXiv preprint, abs/2302.07842.
  16. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. WebGPT: Browser-assis…
  17. OpenAI. 2023. GPT-4 Technical Report.
  18. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large Language Model Connected with Massive APIs. ArXiv preprint, abs/2305.15334.
  19. Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, et al. 2023a. WebCPM: Interactive Web Search for Chinese Long-form Question Answering. ArXiv preprint, abs/2305.06849. find
  20. Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. 2023b. Tool learning with foundation models. ArXiv preprint, abs/2304.08354.
  21. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023c. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs.
  22. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  23. Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. 2023. TPTU: Task planning and tool usage of large language model-based AI agents. ArXiv preprint, abs/2308.03427. — [arXiv:2308.03427](https://arxiv.org/abs/2308.03427) find
  24. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. ArXiv preprint, abs/2302.04761.
  25. Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. 2023. Rest-GPT: Connecting Large Language Models with Real-World Applications via RESTful APIs. ArXiv preprint, abs/2306.06624.
  26. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. 2023. ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases.
  27. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models.
  28. Alan M. Turing. 2009. Computing Machinery and Intelligence, pages 23–65. Springer Netherlands, Dordrecht. find
  29. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. ArXiv preprint, abs/2309.10691.
  30. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
  31. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the Tool Manipulation Capability of Open-source Large Language Models.
  32. Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. 2023a. ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling. ArXiv preprint, abs/2306.11489. find
  33. Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023b. GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction.
  34. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022a. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. In In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS 2022), volume 35, pages 20744–20757. Curran Associates, Inc.
  35. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022b. ReAct: Synergizing reasoning and acting in language models. volume abs/2210.03629.
  36. Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. find
  37. ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios.
  38. Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A Dataset for LLM Question Answering with External Tools. ArXiv preprint, abs/2306.13304.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guo-2024-stabletoolbench-towards-stable-large-scale/