Elamur Research
☰
home / papers / 2023 / FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2305.05176

Summary

Commercial LLM APIs differ widely in price and quality, and running the strongest model on every query quickly becomes costly for high-volume uses such as customer support. The authors examine the pricing of 12 commercial LLMs and find fees that vary by roughly two orders of magnitude. They propose three ways to cut inference cost:

  • Prompt adaptation: use shorter prompts or concatenate queries.
  • LLM approximation: use completion caching or fine-tune a cheap model on outputs from an expensive one.
  • LLM cascade: route each query to an appropriate model.

To demonstrate the cascade idea, they build FrugalGPT. It sends a query to cheaper models first and uses a learned reliability score to decide whether to accept the answer or pass the query to a costlier model such as GPT-4. On HEADLINES, OVERRULING and COQA, FrugalGPT matches GPT-4 at a fraction of the cost and can sometimes exceed its accuracy. This shows that choosing models carefully saves money and energy, even when budget is not the main concern.

Key points

  • On HEADLINES, FrugalGPT matches or exceeds the best single API (GPT-4) with up to 98% lower cost. At equal cost, accuracy improves by about 4% (the abstract's figure; a figure caption says up to 5%).
  • On OVERRULING, it gains 1% accuracy over GPT-4 while cutting cost by 73%.
  • Prices for 10M prompt tokens range from $30 (GPT-4) to $0.2 (GPT-J via Textsynth). Providers covered include OpenAI, AI21, CoHere and Textsynth.
  • Cost rankings depend on the task, because pricing schemes differ. For example, J1 bills only output tokens, while GPT-3 bills both input and output. A pricier API can also perform worse.
  • Limitations: the cascade needs labeled training data drawn from a distribution similar to the test data, and training it carries an upfront cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

178 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 174 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.05176 · arXiv 2305.05176

@misc{chen2023frugalgpt,
  title = {{FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance}},
  author = {Chen, Lingjiao and Zaharia, Matei and Zou, James},
  year = {2023},
  doi = {10.48550/arxiv.2305.05176},
  eprint = {2305.05176},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.05176}
}

References

47 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI21 LLM API. https://www.ai21.com/. Accessed: 2023-03-31. ↗
  2. Simran Arora, Avanika Narayan, Mayee F Chen, Laurel J Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02441, 2022. find
  3. Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91. PMLR, 2018. find
  4. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021. find
  5. Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R Lyu. Towards efficient post-training quantization of pre-trained language models. Advances in Neural Information Processing Systems, 35:1405–1418, 2022. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  7. Stephen Cass. Taking ai to the edge: Google’s tpu now comes in a maker-friendly package. IEEE Spectrum, 56(5):16–17, 2019. find
  8. Lingjiao Chen, Tracy Cai, Matei Zaharia, and James Zou. Did the model change? efficiently assessing machine learning api shifts. arXiv preprint arXiv:2107.14203, 2021. — [arXiv:2107.14203](https://arxiv.org/abs/2107.14203) find
  9. ChatGPT Announcement. https://openai.com/blog/chatgpt. Accessed: 2023-03-31. ↗
  10. Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5918–5926, 2017. find
  11. CoHere LLM API. https://cohere.com/. Accessed: 2023-03-31. Cost estimation of using GPT-3 for real applications. https://www.semianalysis.com/p/the-inference-cost-of-search-disruption. Accessed: 2023-03-31. ↗
  12. Cost estimation of using GPT-3 for real applications. https://neoteric.eu/blog/how-much-does-it-cost-to-use-gpt-models-gpt-3-pricing-explained. Accessed: 2023-03-31. ↗
  13. Lingjiao Chen, Matei Zaharia, and James Y Zou. Frugalml: How to use ml prediction apis more accurately and cheaply. Advances in neural information processing systems, 33:10685–10696, 2020. find
  14. Lingjiao Chen, Matei Zaharia, and James Zou. Efficient online ml api selection for multi-label classification tasks. In International Conference on Machine Learning, pages 3716–3746. PMLR, 2022. find
  15. Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. Successive prompting for decomposing complex questions. arXiv preprint arXiv:2212.04092, 2022. find
  16. Ali Diba, Vivek Sharma, Ali Pazandeh, Hamed Pirsiavash, and Luc Van Gool. Weakly supervised cascaded convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 914–922, 2017. find
  17. forefront AI LLM API. https://beta.forefront.ai/. Accessed: 2023-03-31. ↗
  18. Jerome H Friedman. Stochastic gradient boosting. Computational statistics & data analysis, 38(4):367–378, 2002. find
  19. Ashit Gupta, Anirudh Deodhar, Tathagata Mukherjee, and Venkataramana Runkana. Semi-supervised cascaded clustering for classification of noisy label data. arXiv preprint arXiv:2205.02209, 2022. — [arXiv:2205.02209](https://arxiv.org/abs/2205.02209) find
  20. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015. find
  21. Zhihao Jia, Matei Zaharia, and Alex Aiken. Beyond data and model parallelism for deep neural networks. Proceedings of Machine Learning and Systems, 1:1–13, 2019. find
  22. Eldar Kurtic, Elias Frantar, and Dan Alistarh. Ziplm: Hardware-aware structured pruning of language models. arXiv preprint arXiv:2302.04089, 2023. — [arXiv:2302.04089](https://arxiv.org/abs/2302.04089) find
  23. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689, 2020. find
  24. Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024, 2022. find our card
  25. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022. find
  26. Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. Generated knowledge prompting for commonsense reasoning. arXiv preprint arXiv:2110.08387, 2021. find
  27. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021. find our card
  28. Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. Terapipe: Token-level pipeline parallelism for training large-scale language models. In International Conference on Machine Learning, pages 6543–6552. PMLR, 2021. find
  29. Grégoire Mialon, Roberto Dessı̀, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023. find our card
  30. OpenAI LLM API. https://platform.openai.com/. Accessed: 2023-03-31. ↗
  31. OpenAI. Gpt-4 technical report. arXiv preprint https://arxiv.org/abs/2303.08774, 2023. ↗ our card
  32. Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019. find our card
  33. Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances in neural information processing systems, 24, 2011. find
  34. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. find
  35. Ankur Sinha and Tanmay Khandait. Impact of news on the commodity market: Dataset and results. In Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2, pages 589–601. Springer, 2021. find
  36. Textsynth LLM API. https://textsynth.com/. Accessed: 2023-03-31. ↗
  37. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. find
  38. Paul Viola and Michael J Jones. Robust real-time face detection. International journal of computer vision, 57:137–154, 2004. find
  39. Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021. find
  40. Lidan Wang, Jimmy Lin, and Donald Metzler. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 105–114, 2011. find
  41. Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems, 4:795–813, 2022. find
  42. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. find our card
  43. Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438, 2022. find
  44. Fan Yang, Xuan Li, Qianmu Li, and Tao Li. Exploring the diversity in cluster ensemble generation: Random sampling and random projection. Expert Systems with Applications, 41(10):4844–4866, 2014. find
  45. Zhewei Yao, Cheng Li, Xiaoxia Wu, Stephen Youn, and Yuxiong He. A comprehensive study on post-training quantization for large language models. arXiv preprint arXiv:2303.08302, 2023. — [arXiv:2303.08302](https://arxiv.org/abs/2303.08302) find
  46. Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the eighteenth international conference on artificial intelligence and law, pages… find
  47. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. find our card