Elamur Research
☰
home / papers / 2023 / ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases

ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2306.05301

Summary

Very large models such as GPT-4 can work out how to use tools they have never seen just from their documentation. Smaller open models are usually fine-tuned on a narrow, fixed set of tools and fail on new ones. ToolAlpaca asks whether compact models can learn general tool use without training on each specific tool. The authors build the training data almost automatically. They take short API descriptions from the public-apis list, and an LLM expands each one into standardized documentation: name, introduction, description, function docs and an OpenAPI spec. Next, a multi-agent simulation has LLMs play the user, the assistant and the API itself, which produces multi-turn tool-use dialogues. The resulting corpus has 3938 instances covering more than 400 tools in 50 categories. Fine-tuning Vicuna on it gives ToolAlpaca-7B and ToolAlpaca-13B. Both are tested on simulated and real tools they never saw in training, and the 13B model comes close to GPT-3.5. Experiments also show that the variety of tools matters more than the amount of data. The result offers a low-cost way to give small, self-hosted models broad API-calling ability.

Key points

  • Resource: 3938 simulated tool-use cases built from 400+ APIs in 50 categories. Code and data are released at github.com/tangqiaoyu/ToolAlpaca.
  • Unseen simulated tools (human accept rate): Vicuna-7B scores 16 and ToolAlpaca-7B scores 73. Vicuna-13B scores 25 and ToolAlpaca-13B scores 75. GPT-3.5 scores 79. GPT-4 was used as an automatic judge for procedure, response and overall quality, and its scores agreed well with the human ratings.
  • Real-world APIs (overall accuracy): ToolAlpaca-7B scores 55.3 and ToolAlpaca-13B scores 61.4. The base Vicuna models score 7.9 (7B) and 12.3 (13B), and GPT-3.5 scores 72.8.
  • Out-of-dataset test on GPT4Tools (8 multimodal tools): ToolAlpaca-13B succeeds on 83.7% of cases. GPT4Tools reaches 90.6% but was trained on 71k examples generated the same way as its test set. The 50 test cases that need no tool were excluded.
  • Diversity ablation: with the number of training instances held fixed, raising the toolset from 10 to 400 tools lifts overall accuracy from 51 to 70.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

156 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 151 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.05301 · arXiv 2306.05301

@misc{tang2023toolalpaca,
  title = {{ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases}},
  author = {Tang, Qiaoyu and Deng, Ziliang and Lin, Hongyu and Han, Xianpei and Liang, Qiao and Cao, Boxi and Sun, Le},
  year = {2023},
  doi = {10.48550/arxiv.2306.05301},
  eprint = {2306.05301},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.05301}
}

References

23 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, P.; Alberti, C.; Huot, F.; Maynez, J.; Ma, J.; Ruder, S.; Ganchev, K.; Das, D.; and Lapata, M. 2022. QAmeleon: Multilingual QA with Only 5 Examples. arXiv:2211.08264. Chen, M.; Chen, X.; and tau Yih, W. 2023. Efficient Open Domain Multi-Hop Question Answering with Few-Shot Data Synthesis. a… find
  2. Chen, W.; Ma, X.; Wang, X.; and Cohen, W. W. 2022. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. arXiv:2211.12588. find our card
  3. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. find
  4. Gao, L.; Madaan, A.; Zhou, S.; Alon, U.; Liu, P.; Yang, Y.; Callan, J.; and Neubig, G. 2022. PAL: Program-aided Language Models. arXiv e-prints, arXiv:2211.10435. find
  5. Honovich, O.; Scialom, T.; Levy, O.; and Schick, T. 2022. Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor. arXiv:2212.09689. find our card
  6. Khattab, O.; Santhanam, K.; Li, X. L.; Hall, D.; Liang, P.; Potts, C.; and Zaharia, M. 2023. Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP. arXiv:2212.14024. find our card
  7. Komeili, M.; Shuster, K.; and Weston, J. 2022. Internet-Augmented Dialogue Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8460–8478. Dublin, Ireland: Association for Computational Linguistics. find our card
  8. Li, M.; Song, F.; Yu, B.; Yu, H.; Li, Z.; Huang, F.; and Li, Y. 2023. API-Bank: A Benchmark for Tool-Augmented LLMs. arXiv:2304.08244. — [arXiv:2304.08244](https://arxiv.org/abs/2304.08244) find
  9. Lu, P.; Peng, B.; Cheng, H.; Galley, M.; Chang, K.-W.; Wu, Y. N.; Zhu, S.-C.; and Gao, J. 2023. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. arXiv:2304.09842. Mialon, G.; Dessı̀, R.; Lomeli, M.; Nalmpantis, C.; Pasunuru, R.; Raileanu, R.; Rozière, B.; Schick, T.; Dwi… find our card
  10. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774. find our card
  11. Parisi, A.; Zhao, Y.; and Fiedel, N. 2022. TALM: Tool Augmented Language Models. CoRR, abs/2205.12255. find our card
  12. Patil, S. G.; Zhang, T.; Wang, X.; and Gonzalez, J. E. 2023. Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334. find our card
  13. Qin, Y.; Hu, S.; Lin, Y.; Chen, W.; Ding, N.; Cui, G.; Zeng, Z.; Huang, Y.; Xiao, C.; Han, C.; Fung, Y. R.; Su, Y.; Wang, H.; Qian, C.; Tian, R.; Zhu, K.; Liang, S.; Shen, X.; Xu, B.; Zhang, Z.; Ye, Y.; Li, B.; Tang, Z.; Yi, J.; Zhu, Y.; Dai, Z.; Yan, L.; Cong, X.; Lu, Y.; Zhao, W.; Huang, Y.; Yan,… find our card
  14. Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; Zhao, S.; Tian, R.; Xie, R.; Zhou, J.; Gerstein, M.; Li, D.; Liu, Z.; and Sun, M. 2023b. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv:2307.16789. Schick, T.; Dwivedi-Yu… find our card
  15. Shen, Y.; Song, K.; Tan, X.; Li, D.; Lu, W.; and Zhuang, Y. 2023. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. arXiv:2303.17580. find our card
  16. Song, Y.; Xiong, W.; Zhu, D.; Li, C.; Wang, K.; Tian, Y.; and Li, S. 2023. RestGPT: Connecting Large Language Models with Real-World Applications via RESTful APIs. arXiv:2306.06624. find
  17. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https: //github.com/tatsu-lab/stanford alpaca. find
  18. Wang, S.; Liu, Y.; Xu, Y.; Zhu, C.; and Zeng, M. 2021. Want To Reduce Labeling Cost? GPT-3 Can Help. In Findings of the Association for Computational Linguistics: EMNLP 2021, 4195–4205. Punta Cana, Dominican Republic: Association for Computational Linguistics. find our card
  19. Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv:2212.10560. find
  20. Xu, Q.; Hong, F.; Li, B.; Hu, C.; Chen, Z.; and Zhang, J. 2023. On the Tool Manipulation Capability of Open-source Large Language Models. arXiv:2305.16504. find
  21. Yang, R.; Song, L.; Li, Y.; Zhao, S.; Ge, Y.; Li, X.; and Shan, Y. 2023. GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction. arXiv:2305.18752. find
  22. Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2022. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv preprint arXiv:2210.03629. find our card
  23. Zhou, C.; Liu, P.; Xu, P.; Iyer, S.; Sun, J.; Mao, Y.; Ma, X.; Efrat, A.; Yu, P.; Yu, L.; Zhang, S.; Ghosh, G.; Lewis, M.; Zettlemoyer, L.; and Levy, O. 2023. LIMA: Less Is More for Alignment. arXiv:2305.11206. find