Elamur Research
☰
home / papers / 2023 / The False Promise of Imitating Proprietary LLMs

The False Promise of Imitating Proprietary LLMs

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2305.15717

Summary

Many open-source chat models, including Alpaca, Vicuna and Koala, are built by fine-tuning a weaker model on ChatGPT outputs. Some of them claim to be close to proprietary systems. This paper tests whether that kind of imitation really transfers the capabilities. The authors collected two kinds of imitation data from ChatGPT: task-specific and broad-coverage. They used it to fine-tune GPT-2 and LLaMA base models with 1.5B–13B parameters, on 0.3M–150M tokens of data. They compared the resulting models with ChatGPT in two ways: blind pairwise judgments by crowdworkers and GPT-4, and benchmarks (MMLU, Natural Questions, HumanEval). Human raters liked the imitators. The benchmarks, however, showed almost no narrowing of the gap to ChatGPT, except on tasks well covered by the imitation data. The models pick up ChatGPT's confident, well-structured style but still make factual errors that non-expert raters miss. Using a larger base model helped much more than adding imitation data. The study warns against judging models only by human preference. It argues that stronger pretrained base LMs, not imitation shortcuts, are what will improve open models.

Key points

  • About 70% of the best imitation model's outputs were rated by crowdworkers as equal to or better than ChatGPT's. Targeted automatic evaluations did not support this.
  • Training on around 100k ChatGPT responses to broad user queries gave no gain on Natural Questions, and more data even made it worse. Imitation data built from Natural-Questions-style queries, by contrast, improved accuracy sharply.
  • Making the LLaMA base model larger improved results far more than adding more imitation data. The authors see weak base capabilities as the main limit of open LMs.
  • Imitation improves style, persona and instruction following, but not factuality, coding or problem solving. It is still useful for avoiding costly fine-tuning annotation when the base LM is already strong.
  • Implications:
    • Companies can protect their lead by pretraining strong base models.
    • Crowdworker evaluation has trouble telling imitators apart from the original.
    • Imitation copies the target model's biases, and its quality is roughly capped at the target's level.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

130 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 118 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.15717 · arXiv 2305.15717

@misc{gudibande2023false,
  title = {{The False Promise of Imitating Proprietary LLMs}},
  author = {Gudibande, Arnav and Wallace, Eric and Snell, Charlie and Geng, Xinyang and Liu, Hao and Abbeel, Pieter and Levine, Sergey and Song, Dawn},
  year = {2023},
  doi = {10.48550/arxiv.2305.15717},
  eprint = {2305.15717},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.15717}
}

References

46 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. OpenAI. ChatGPT: Optimizing language models for dialogue., 2022. find
  2. Sundar Pichai. An important next step on our AI journey. Google AI Blog, 2023. find
  3. AnthropicAI. Introducing claude, 2023. find
  4. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMa: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. find
  5. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. find
  6. Eric Wallace, Mitchell Stern, and Dawn Song. Imitation attacks and defenses for black-box machine translation systems. In EMNLP, 2020. find
  7. Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In CVPR, 2019. find
  8. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning Workshop, 2014. find our card
  9. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022a. find
  10. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model, 2023. find
  11. Dylan Patel and Afzal Ahmad. Google “we have no moat, and neither does OpenAI”, 2023. find
  12. Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction APIs. In USENIX Security Symposium, 2016. find
  13. Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. Is Chat-GPT good at search? investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09542, 2023. find our card
  14. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301, 2023. find our card
  15. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689, 2022. find our card
  16. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* chatgpt quality, 2023. find
  17. Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. BAIR Blog, 2023. find
  18. Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. GPT4All: Training an assistant-style chatbot with large scale data distillation from GPT-3.5-Turbo, 2023. find
  19. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv preprint arXiv:2304.03277, 2023. find our card
  20. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Change, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural quest… find our card
  21. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597, 2023. find our card
  22. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. In EMNLP, 2022b. find
  23. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. In OpenAI Technical Report, 2019. find
  24. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. find
  25. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021. find our card
  26. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: A benchmark for question answering research. TACL, 2019b. find our card
  27. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. find our card
  28. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of EMNLP, 2020. find our card
  29. John Schulman. Reinforcement learning from human feedback: Progress and challenges. 2023. find
  30. Leo Gao. Behavior cloning is miscalibrated. Alignment Forum, 2021. find
  31. Yoav Goldberg. Reinforcement learning for language models. 2023. find
  32. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NIPS, 2017. find
  33. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022. find our card
  34. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022. find our card
  35. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023. find our card
  36. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mPLUG-Owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. find our card
  37. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. find our card
  38. Daniel Lowd and Christopher Meek. Adversarial learning. In KDD, 2005. find
  39. Kalpesh Krishna, Gaurav Singh Tomar, Ankur P Parikh, Nicolas Papernot, and Mohit Iyyer. Thieves on sesame street! Model extraction of BERT-based APIs. In ICLR, 2020. find
  40. Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy. A framework for the extraction of deep neural networks by leveraging public data. arXiv preprint arXiv:1905.09165, 2019. — [arXiv:1905.09165](https://arxiv.org/abs/1905.09165) find
  41. Mika Juuti, Sebastian Szyller, Samuel Marchal, and N Asokan. PRADA: protecting against DNN model stealing attacks. In IEEE EuroS&P, 2019. find
  42. Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N Asokan. DAWN: Dynamic adversarial watermarking of neural networks. In ACM Multimedia, 2019. find
  43. Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. In ICLR, 2021. find
  44. Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Towards defenses against DNN model stealing attacks. In ICLR, 2020. find
  45. Adam Dziedzic, Nikita Dhawan, Muhammad Ahmad Kaleem, Jonas Guan, and Nicolas Papernot. On the difficulty of defending self-supervised learning against model extraction. In ICLR, 2022a. find
  46. Adam Dziedzic, Muhammad Ahmad Kaleem, Yu Shen Lu, and Nicolas Papernot. Increasing the cost of model extraction with calibrated proof of work. In ICLR, 2022b. find