Elamur Research⌕
☰
home / papers / 2023 / MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use

MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use

Paper · 2023 · Yue Huang, Jiawen Shi, Yuan Li et al.

Summary

Most earlier tool-use benchmarks for LLMs test how a model fills in tool arguments and handles tool outputs. MetaTool targets the two decisions that come first in a ReAct-style agent loop: whether a query needs an external tool at all, and which tool or tools to pick from a candidate list. The authors built ToolE, a set of roughly 21k user queries tied to tools taken from OpenAI's plugin list. The queries cover both single-tool and multi-tool requests. To make the phrasing more varied, they generated queries with several prompting strategies. Tools with overlapping functions were merged or split so that each query has an unambiguous answer, and the data was checked by humans. On top of ToolE they defined an awareness test and four tool-selection subtasks. Testing eight popular LLMs showed that most cannot judge the limits of their own abilities and choose tools unreliably, and that performance varies across everyday scenarios. Error analysis suggests there is still room for improvement. The benchmark matters for anyone building agents (AutoGPT, MetaGPT-style) that rely on tool catalogs.

Key points

  • ToolE dataset: 21,127 queries built from OpenAI plugin names and descriptions. Generation used four methods: emotional, keyword, direct diverse and detailed. Overlapping tools were resolved through merging and decomposition.
  • Awareness test set: combines ToolE queries with existing instruction datasets. The model answers yes/no on whether it needs a tool.
  • Four selection subtasks:
    • choosing among similar tools (semantic understanding);
    • selection in specific scenarios, e.g., different user groups (adaptability);
    • selection where the correct answer may be "None" (reliability/hallucination);
    • choosing up to two tools (multi-tool reasoning).
  • Eight LLMs evaluated, including ChatGPT, GPT-4, Llama2 and Vicuna. Failure cases show three patterns:
    • overconfidence, such as a model claiming it can set reminders or browse the web;
    • needless reliance on tools for plain language tasks;
    • contradictory self-assessment on math queries.
  • Advice for tool developers: pick the model that rewrites tool descriptions according to the downstream LLM that will consume them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom256 works · 123,373 citations🎓Duke UniversityDurham · United States37 works · 15,861 citations🎓Huazhong University of Science and TechnologyWuhan · China59 works · 8,310 citations🎓Lehigh UniversityBethlehem · United States10 works · 1,573 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong and Lichao Sun. 2023. MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use. arXiv:2310.03128.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.03128 · arXiv 2310.03128

@misc{huang2023metatool,
  title = {{MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use}},
  author = {Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong and Lichao Sun},
  year = {2023},
  doi = {10.48550/arxiv.2310.03128},
  eprint = {2310.03128},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.03128}
}

References

91 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nomic AI. Nomic ai, 2023. https://atlas.nomic.ai/. ↗
  2. Abid Ali Awan. The 10 best chatgpt plugins for data science, 2023. https://www.datacamp.com/blog/the-10-best-chat-gpt-plugins-for-data-science. babyagi. ↗
  3. Babyagi, 2023. https://github.com/yoheinakajima/babyagi. ↗
  4. Baichuan. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. URL https://arxiv.org/abs/2309.10305. ↗
  5. NATALY BIRCH and ANDRIAN VALEANU. 16 best ai tools for web designers, 2023. https://designmodo.com/ai-tools-designers/. ↗
  6. Adrian Bridgwater. Auto-tech series - octoml: Large language model (llm) automation for developers, 2023. https://www.computerweekly.com/blog/CW-Developer-Network/Auto-tech-series-OctoML-Large-Language-Model-LLM-automation-for-developers. ↗
  7. Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXiv:2305.17126, 2023.
  8. Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S Yu, and Lichao Sun. A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt. arXiv preprint arXiv:2303.04226, 2023. find
  9. Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. ArXiv, abs/2305.04160, 2023. URL https://api.semanticscholar.org/CorpusID:258558106. — [arXiv:2305.04160](https://arxiv.org/abs/2305.04160) ↗
  10. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna… ↗
  11. Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712, 2022. find
  12. Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. Uncovering chatgpt’s capabilities in recommender systems. arXiv preprint arXiv:2305.02182, 2023. — [arXiv:2305.02182](https://arxiv.org/abs/2305.02182) find
  13. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.06070, 2023.
  14. Empresaria. House manager job description, 2023. https://www.greycoatlumleys.co.uk/looking-for-a-job/career-advice/house-manager-job-description. ↗
  15. Emilio Ferrara. Should chatgpt be biased? challenges and risks of bias in large language models. arXiv preprint arXiv:2304.03738, 2023. find
  16. Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. Chat-rec: Towards interactive and explainable llms-augmented recommender system. arXiv preprint arXiv:2303.14524, 2023. geekan. — [arXiv:2303.14524](https://arxiv.org/abs/2303.14524) find
  17. Metagpt, 2023. https://github.com/geekan/MetaGPT. ↗
  18. Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL https://bair.berkeley.edu/blog/2023/04/03/koala/. ↗
  19. Riley Goodside. The past and future of prompt engineering, 2023. https://exchange.scale.com/public/videos/the-past-and-future-of-prompt-engineering-2023-08-23. ↗
  20. GPTStore. Overview of ai/chatgpt plugin weather, 2023. weather — vicentescode-repl-co. find
  21. Level Up Coding Youssef Hosni. 9 helpful chatgpt plugins for data scientists, 2023. https://levelup.gitconnected.com/9-helpful-chatgpt-plugins-for-data-scientists-32eceb8d07a8. ↗
  22. Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. arXiv preprint arXiv:2308.00675, 2023. find
  23. Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. Chatdb: Augmenting llms with databases as their symbolic memory. arXiv preprint arXiv:2306.03901, 2023. — [arXiv:2306.03901](https://arxiv.org/abs/2306.03901) find
  24. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023.
  25. Vartika Kashyap. 26 best designer tools for web and graphic design professionals, 2023. https://www.proofhub.com/articles/designer-tools. ↗
  26. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103…
  27. Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115, 2022.
  28. Cheng Li, Jindong Wang, Kaijie Zhu, Yixuan Zhang, Wenxin Hou, Jianxun Lian, and Xing Xie. Emotionprompt: Leveraging psychology for large language models enhancement via emotional stimulus. arXiv preprint arXiv:2307.11760, 2023a.
  29. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. ArXiv, abs/2306.00890, 2023b. URL https://api.semanticscholar.org/CorpusID:258999820… ↗
  30. Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. arXiv preprint arXiv:2305.03111, 2023c.
  31. Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244, 2023d. — [arXiv:2304.08244](https://arxiv.org/abs/2304.08244) find
  32. Yuan Li, Yixuan Zhang, and Lichao Sun. Metaagents: Simulating interactions of human behaviors for llm-based task-oriented coordination via collaborative generative agents, 2023e. find
  33. Gemingtian Liu, Xinyu Ma, Yu Zhang, Boyan Su, and Pinan Liu. Gpt4: The indispensable helper for neurosurgeons in the new era. Annals of Biomedical Engineering, 51:2113 – 2115, 2023. URL https://api.semanticscholar.org/CorpusID:258787338. ↗
  34. Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. ArXiv, abs/2306.09093, 2023. URL https://api.semanticscholar.org/CorpusID: 259165461. https://gptsto… ↗
  35. Teresa Mears. 10 essential tech tools for older adults, 2015. https://money.usnews.com/money/retirement/articles/2015/11/16/10-essential-tech-tools-for-older-adults. ↗
  36. Ofer Mendelevitch. Large language models for code generation – part 2, 2023. https://vectara.com/large-language-models-llms-for-code-generation-part-2/. ↗
  37. Manuel Odendahl. Llms will fundamentally change software engineering, 2023. https://dev.to/wesen/llms-will-fundamentally-change-software-engineering-3oj8. ↗
  38. OpenAI. Introducing chatgpt, 2023a. URL https://openai.com/blog/chatgpt. ↗
  39. OpenAI. Gpt-4 technical report. 2023b. URL https://arxiv.org/pdf/2303.08774.pdf. ↗
  40. OpenAI. new-and-improved-embedding-model, new-and-improved-embedding-model. OpenAI. Openai plugin, 2023d. https://openai.com/blog/chatgpt-plugins. ↗
  41. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. find
  42. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.
  43. Zhi Qi, Yi Yu, Meiqi Tu, Junyi Tan, and Yongfeng Huang. Foodgpt: A large language model in food testing domain with incremental pre-training and knowledge graph prompt. ArXiv, abs/2308.10173, 2023. URL https://api.semanticscholar.org/CorpusID:261048937. — [arXiv:2308.10173](https://arxiv.org/abs/2308.10173) ↗
  44. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023.
  45. Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint arXiv:2304.08354, 2023a.
  46. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023b.
  47. Salvatore Raieli. Fingpt: open-source llm for finance, 2023. https://levelup.gitconnected.com/fingpt-open-source-llm-for-finance-e8ec10d0bf40. ↗
  48. Rapid. Rapid api, 2023. https://rapidapi.com/. ↗
  49. Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53–65, 1987. find
  50. Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. Tptu: Task planning and tool usage of large language model-based ai agents. arXiv preprint arXiv:2308.03427, 2023. — [arXiv:2308.03427](https://arxiv.org/abs/2308.03427) find
  51. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  52. National Science and Technology Council. Emerging technologies to support an aging population, 2019. https://trumpwhitehouse.archives.gov/wp-content/uploads/2019/03/Emerging-Tech-to-Support-Aging-2019.pdf. ↗
  53. ShareGPT. Sharegpt, 2023. https://sharegpt.com/. ↗
  54. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580, 2023. 2023c. https://openai.com/blog/ ↗
  55. Significant-Gravitas. Autogpt, 2023. https://github.com/Significant-Gravitas/Auto-GPT. ↗
  56. Peter HA Sneath, Robert R Sokal, et al. Numerical taxonomy. The principles and practice of numerical classification. 1973. find
  57. Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang… — TrustLLM: Trustworthiness in Large Language Models
  58. Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09542, 2023.
  59. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
  60. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023.
  61. ODSC Team. Have you met fingpt? a new open-source financial large language model, 2023. https://opendatascience.com/have-you-met-fingpt-a-new-open-source-financial-large-language-model/. ↗
  62. THUDM. Chatglm2, 2023. https://github.com/THUDM/ChatGLM2-6B. ↗
  63. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  64. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. find
  65. Nicolas Vidal. How ai and llms are streamlining financial services, 2023. https://www.forbes.com/sites/forbestechcouncil/2023/05/05/how-ai-and-llms-are-streamlining-financial-services/?sh=2ea8b923017a. ↗
  66. Leandro von Werra and Loubna Ben Allal. Starcoder: A state-of-the-art llm for code, 2023. https://huggingface.co/blog/starcoder. ↗
  67. Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, and Xiaohu Li. Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation. ArXiv, abs/2306.09968, 2023a. URL https://api.semanticscholar.org/CorpusID:259187929. ↗
  68. Hao Wang, Chi-Liang Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. Huatuo: Tuning llama model with chinese medical knowledge. ArXiv, abs/2304.06975, 2023b. URL https://api.semanticscholar.org/CorpusID:258170497. ↗
  69. Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975, 2023c.
  70. Wen Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Y. Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. ArXiv, abs/2305.11175, 2023d. URL https://api.semanticscholar.org/CorpusID: 258762579. — [arXiv:2305.11175](https://arxiv.org/abs/2305.11175) ↗
  71. Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. Rethinking the evaluation for conversational recommendation in the era of large language models. arXiv preprint arXiv:2305.13112, 2023e. find
  72. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022. find
  73. Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V Le. Simple synthetic data reduces sycophancy in large language models. arXiv preprint arXiv:2308.03958, 2023.
  74. Matt Welsh. The future of software development with llms is here: Announcing fixie’s developer preview and 17m in seed funding, 2023. https://blog.fixie.ai/the-future-of-software-development-with-llms-is-here-announcing-fixies-developer-preview-and-17m-cf6fca0c4041. ↗
  75. Stephen Wolfram Writings. Chatgpt gets its “wolfram superpowers”!, 2023. https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its-wolfram-superpowers/. ↗
  76. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023a.
  77. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023b.
  78. Daniel Xiao. Ai-based literature review tools, 2023. https://tamu.libguides.com/c.php?g=1289555. ↗
  79. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504, 2023.
  80. Zhiling Yan, Kai Zhang, Rong Zhou, Lifang He, Xiang Li, and Lichao Sun. Multimodal chatgpt for medical applications: an experimental study of gpt-4v. arXiv preprint arXiv:2310.19061, 2023. — [arXiv:2310.19061](https://arxiv.org/abs/2310.19061) find
  81. Hongyang Yang, Xiao-Yang Liu, and Chris Wang. Fingpt: Open-source financial large language models. ArXiv, abs/2306.06031, 2023a. URL https://api.semanticscholar.org/CorpusID:259129734. ↗
  82. Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031, 2023b.
  83. Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. arXiv preprint arXiv:2305.18752, 2023c.
  84. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
  85. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015, 2023a. URL https://arxiv.org/abs/2304.02015. ↗
  86. Zhengqing Yuan, Zhaoxu Li, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.16862, 2023b. — [arXiv:2312.16862](https://arxiv.org/abs/2312.16862) find
  87. Jennifer Zhang. Unleashing the potential of llms: a new era for financial services, 2023. https://www.wiz.ai/unleashing-the-potential-of-llms-a-new-era-for-financial-services/. ↗
  88. Kai Zhang, Jun Yu, Zhiling Yan, Yixin Liu, Eashan Adhikarla, Sunyang Fu, Xun Chen, Chen Chen, Yuyin Zhou, Xiang Li, et al. Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks. arXiv preprint arXiv:2305.17100, 2023. find
  89. Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419, 2023a. find
  90. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023b. find
  91. Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2023-metatool-benchmark-large-language-models/