Elamur Research⌕
☰
home / papers / 2024 / ToolACE: Winning the Points of LLM Function Calling

ToolACE: Winning the Points of LLM Function Calling

Paper · 2024 · Weiwen Liu; Xu Huang; Xingshan Zeng; Xinlong Hao; Shuai Yu; Dexun Li; Shuai Wang; Weinan Gan; Zhengying Liu; Yuanqing Yu; Zezhong Wang; Yuxian Wang; Wu Ning; Yutai Hou; Bin Wang; Chuhan Wu; Xinzhi Wang; Yong Liu; Yasheng Wang; Duyu Tang; Dandan Tu; Lifeng Shang; Xin Jiang; Ruiming Tang; Defu Lian; Qun Liu; and Enhong Chen

Summary

Teaching LLMs to call external functions requires training data that is both varied and correct. Real annotated examples are scarce, and existing synthetic pipelines tend to rely on limited public APIs, cover few domains, and focus on simple single-turn calls. ToolACE is an automated, agent-driven pipeline for producing such data. Its first component, Tool Self-Evolution Synthesis (TSS), mines API-related documents from LLM pretraining data to build a hierarchical tree of domains and functionalities. It then grows a large pool of synthetic APIs through repeated mutation of parameters, types and constraints. Next, Self-Guided Dialog Generation (SDG) has user, assistant and tool agents simulate conversations, while the model being fine-tuned rates complexity so that examples sit just above its current ability. Finally, Dual-Layer Validation (DLV) filters the output with rule-based and model-based checks. An 8B-parameter model trained on this data reaches state-of-the-art results on BFCL and API-Bank, rivaling recent GPT-4 models. This suggests that careful data design can offset model scale.

Key points

  • The API pool contains 26,507 APIs across 390 domains, more than Gorilla, ToolAlpaca, ToolLLM or xLAM. It supports nested parameters such as lists of dictionaries.
  • Four dialog types are generated: single, parallel and dependent function calls, plus non-tool-use dialogs.
  • Rule checkers verify types, value constraints and format. Model checkers assess consistency, and human validation is also part of the pipeline.
  • The model was evaluated on BFCL and API-Bank. With 8B parameters, it beats open-source LLMs and is competitive with GPT-4.
  • The model and part of the dataset are released on Hugging Face under Team-ACE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🎓University of Science and Technology of ChinaHefei · China145 works · 114,409 citations🎓Chinese University of Hong KongHong Kong · China159 works · 16,154 citations🏢Huawei TechnologiesShenzhen · China96 works · 15,432 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Weiwen Liu; Xu Huang; Xingshan Zeng; Xinlong Hao; Shuai Yu; Dexun Li; Shuai Wang; Weinan Gan; Zhengying Liu; Yuanqing Yu; Zezhong Wang; Yuxian Wang; Wu Ning; Yutai Hou; Bin Wang; Chuhan Wu; Xinzhi Wang; Yong Liu; Yasheng Wang; Duyu Tang; Dandan Tu; Lifeng Shang; Xin Jiang; Ruiming Tang; Defu Lian; Qun Liu; and Enhong Chen. 2024. ToolACE: Winning the Points of LLM Function Calling. arXiv:2409.00920.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.00920 · arXiv 2409.00920

@misc{liu2024toolace,
  title = {{ToolACE: Winning the Points of LLM Function Calling}},
  author = {Weiwen Liu; Xu Huang; Xingshan Zeng; Xinlong Hao; Shuai Yu; Dexun Li; Shuai Wang; Weinan Gan; Zhengying Liu; Yuanqing Yu; Zezhong Wang; Yuxian Wang; Wu Ning; Yutai Hou; {Bin Wang; Chuhan Wu; Xinzhi Wang; Yong Liu; Yasheng Wang; Duyu Tang; Dandan Tu; Lifeng Shang; Xin Jiang; Ruiming Tang; Defu Lian; Qun Liu;} and Enhong Chen},
  year = {2024},
  doi = {10.48550/arxiv.2409.00920},
  eprint = {2409.00920},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.00920}
}

References

34 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Bhargav, Maxwell Crouse, Chulaka Gunasekara, et al. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. arXiv preprint arXi… find
  2. AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md. ↗
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — Qwen Technical Report
  4. Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A Lastras. Api-blend: A comprehensive corpora for training and benchmarking api llms. arXiv preprint arXiv:2402.15491, 2024. find
  5. André Bauer, Simon Trapp, Michael Stenger, Robert Leppich, Samuel Kounev, Mark Leznik, Kyle Chard, and Ian Foster. Comprehensive exploration of synthetic data generation: A survey. arXiv preprint arXiv:2401.02524, 2024. — [arXiv:2401.02524](https://arxiv.org/abs/2401.02524) find
  6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  8. Qianlong Du, Chengqing Zong, and Jiajun Zhang. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311.15653, 2023. — [arXiv:2311.15653](https://arxiv.org/abs/2311.15653) find
  9. Yilun Hao, Yongchao Chen, Yang Zhang, and Chuchu Fan. Large language models can plan your travels rigorously with formal verification tools. arXiv preprint arXiv:2404.11891, 2024. — [arXiv:2404.11891](https://arxiv.org/abs/2404.11891) find
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
  11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
  12. Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models, 2023. find
  13. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. ↗
  14. Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, et al. Planning, creation, usage: Benchmarking llms for comprehensive tool utilization in real-world complex scenarios. arXiv preprint arXiv:2401.17167, 2024. find
  15. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244, 2023.
  16. Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406.18518, 2024. — [arXiv:2406.18518](https://arxiv.org/abs/2406.18518) find
  17. Meetkai. Functionary.meetkai. 2024. URL https://functionary.meetkai.com. ↗
  18. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey, 2023.
  19. Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, et al. Agentinstruct: Toward generative teaching with agentic flows. arXiv preprint arXiv:2407.03502, 2024. find
  20. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.
  21. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023.
  22. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. arXiv preprint arXiv:2405.17935, 2024.
  23. Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Ziyue Li, Xingyu Zeng, and Rui Zhao. Tptu: Large language model-based ai agents for task planning and tool usage, 2023. find
  24. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023.
  25. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu…
  26. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023.
  27. Adrian Theuma and Ehsan Shareghi. Equipping language models with tool use capability for tabular data analysis in finance. arXiv preprint arXiv:2401.15328, 2024. find
  28. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023. find
  29. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023.
  30. Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024. ↗
  31. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
  32. Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2023.
  33. Dylan Zhang, Justin Wang, and Francois Charton. Instruction diversity drives generalization to unseen tasks. arXiv preprint arXiv:2402.10891, 2024. — [arXiv:2402.10891](https://arxiv.org/abs/2402.10891) find
  34. Ruizhe Zhong, Xingbo Du, Shixiong Kai, Zhentao Tang, Siyuan Xu, Hui-Ling Zhen, Jianye Hao, Qiang Xu, Mingxuan Yuan, and Junchi Yan. Llm4eda: Emerging progress in large language models for electronic design automation. arXiv preprint arXiv:2401.12224, 2023. — [arXiv:2401.12224](https://arxiv.org/abs/2401.12224) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2024-toolace-winning-points-llm-function/