Elamur Research⌕
☰
home / papers / 2023 / RestGPT: Connecting Large Language Models with Real-World RESTful APIs

RestGPT: Connecting Large Language Models with Real-World RESTful APIs

Summary

Most tool-augmented LLMs work with a small set of custom-built tools and struggle with multi-step requests, so they fall short in realistic settings. This paper links LLMs to RESTful APIs, the dominant style for web services, which are documented through the OpenAPI Specification (OAS). The proposed framework, RestGPT, splits work among three prompted LLM modules. A Planner proposes the next sub-task in natural language. An API Selector turns that sub-task into concrete API calls. An Executor then runs the calls through two parts: a Caller, which builds parameters from the API docs, and a Parser, which writes Python code from the OAS response schema to extract data from the JSON output. Planning happens online and iteratively, so the system revises its next step after seeing each execution result. The authors also release RestBench, a benchmark of human-written instructions with annotated gold solution paths in two domains: the TMDB movie database and the Spotify music player. RestGPT beats offline plan-then-execute and ReAct baselines, especially on long tasks. This suggests LLM agents can plug into existing web applications without hand-built tool wrappers.

Key points

  • RestGPT supports over 100 RESTful APIs. Earlier systems use far fewer, such as the 15 tools in Chameleon. Because RestGPT works with any OAS-described service, it can be added to an application without custom integration.
  • Each module reads only the part of the OAS it needs: the selector uses endpoint descriptions, the Caller uses detailed docs, and the Parser uses response schemas. This helps fit large API sets into the LLM's limited context window.
  • On instructions whose gold path has length 4, all baselines largely fail, while RestGPT still succeeds more than 40% of the time.
  • In a scaling test, the authors started with 10 TMDB APIs and 15 instructions, then added irrelevant "noise" APIs. Baseline performance dropped as noise grew, but RestGPT stayed nearly unaffected.
  • In a Spotify playlist case study, RestGPT completed the task in 4 API calls versus 6 for ReAct. The offline planner picked a wrong API and ignored dependencies between calls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

91 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 84 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.06624 · arXiv 2306.06624

@misc{song2023restgpt,
  title = {{RestGPT: Connecting Large Language Models with Real-World RESTful APIs}},
  author = {Song, Yifan and Xiong, Weimin and Zhu, Dawei and Wu, Wenhao and Qian, Han and Song, Mingbo and Huang, Hailiang and Li, Cheng and Wang, Ke and Yao, Rong and Tian, Ye and Li, Sujian},
  year = {2023},
  doi = {10.48550/arxiv.2306.06624},
  eprint = {2306.06624},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.06624}
}

References

26 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  2. OpenAI. Chatgpt, 2022. URL https://openai.com/blog/chatgpt. ↗
  3. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022. find our card
  4. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. find our card
  5. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022. find our card
  6. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118–9147. PMLR, 2022. find our card
  7. Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023. find our card
  8. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023. find our card
  9. Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint arXiv:2304.08354, 2023. find our card
  10. Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022. find our card
  11. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. find our card
  12. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842, 2023. find our card
  13. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580, 2023. find our card
  14. Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023. find our card
  15. Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. arXiv preprint arXiv:2305.18752, 2023. find
  16. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. find our card
  17. Li Li, Wu Chou, Wei Zhou, and Min Luo. Design patterns and extensibility of rest api for networking applications. IEEE Transactions on Network and Service Management, 13(1):154–167, 2016. find
  18. SmartBear. Swagger, 2023. URL https://swagger.io/. ↗
  19. Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014, 2023. find
  20. Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128, 2023. find our card
  21. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023. find our card
  22. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023. find our card
  23. Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244, 2023. — [arXiv:2304.08244](https://arxiv.org/abs/2304.08244) find
  24. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023. find
  25. Mark Masse. REST API design rulebook: designing consistent RESTful web service interfaces. " O’Reilly Media, Inc.", 2011. find
  26. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366, 2023. *Figure 6: A RESTful API from TMDB.* find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/song-2023-restgpt-connecting-large-language-models/