Elamur Research
☰
home / papers / 2023 / Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data

Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2304.01196

Summary

In 2023, the strongest chat models, such as ChatGPT, were available only through restricted APIs. There were also few good public corpora of multi-turn dialogue for training open alternatives. The authors build such a corpus cheaply with "self-chat": ChatGPT (gpt-3.5-turbo) writes both the user and the assistant turns. A fixed template and a seed question set the topic of each dialogue. Seeds come from Quora, Stack Overflow, or the medical MedQuAD dataset. They then fine-tune LLaMA with LoRA, which trains only a small set of added low-rank weights. The result is the Baize family: 7B, 13B and 30B models, plus Baize-Healthcare. Version 1.5 has a separate ChatGPT call regenerate each assistant turn, which gives longer and more detailed answers. Version 2 adds Self-Distillation with Feedback (SDF), a cheaper alternative to RLHF that uses ChatGPT's judgments of Baize's own outputs for further tuning. The models hold multi-turn conversations, refuse unethical requests, and run on a single GPU. Together, the method, data and weights give a cheap, reproducible way to build open and domain-specific chat assistants.

Key points

  • Data cost: The v1 corpus has 111.5k self-chat dialogues, about 55k seeded from each of Quora and Stack Overflow, and cost roughly $100 in API calls. MedQuAD seeds added 47k medical dialogues for Baize-Healthcare, and Stanford Alpaca data was mixed in to improve instruction following.
  • Dialogue shape: Self-chat dialogues average 3.6–3.9 turns, while Alpaca is single-turn. In the v2-style data (used for v1.5), average response length rises from about 36 to 149.6 (Quora v2) and 78.2 (Stack Overflow v2).
  • Training budget: LoRA trains only 17.9M parameters for 7B, 28.0M for 13B and 54.6M for 30B. Training v1 took 9–36 GPU-hours on one A100-80G. v1.5 used 8 GPUs (32 and 64 GPU-hours), and SDF for v2 took 38 and 76 GPU-hours.
  • Limitations: The GPT-4-based automatic evaluation favours longer answers and has a positional bias. Baize inherits LLaMA's outdated knowledge (at least 82% of pretraining data is from before 2020). LLaMA covers only 20 languages and has thin non-English data, which matters for Ukrainian use.
  • Licensing: The weights are for research use only and are subject to Meta's LLaMA license. The corpus is released under CC-BY-NC 4.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

171 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 153 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.01196 · arXiv 2304.01196

@misc{xu2023baize,
  title = {{Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data}},
  author = {Xu, Canwen and Guo, Daya and Duan, Nan and McAuley, Julian},
  year = {2023},
  doi = {10.48550/arxiv.2304.01196},
  eprint = {2304.01196},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.01196}
}

References

29 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977. find our card
  2. Asma Ben Abacha and Dina Demner-Fushman. 2019. A question-entailment approach to question answering. BMC bioinformatics, 20(1):1–23. find
  3. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality. https://vicuna.lmsys.org/. ↗
  4. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. find
  5. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. find
  6. Demi Guo, Alexander M. Rush, and Yoon Kim. 2021. Parameter-efficient transfer learning with diff pruning. In ACL-IJCNLP, pages 4884–4896. Association for Computational Linguistics. find our card
  7. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In ICLR. OpenReview.net. find our card
  8. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In ICLR. OpenReview.net. find
  9. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 2790–2799. PMLR. find
  10. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net. find
  11. Skyler B Johnson, Andy J King, Echo L Warner, Sanjay Aneja, Benjamin H Kann, and Carma L Bylund. 2023. Using chatgpt to evaluate cancer myths and misconceptions: artificial intelligence and cancer information. JNCI Cancer Spectrum, 7(2):pkad015. find
  12. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL-IJCNLP, pages 4582–4597. Association for Computational Linguistics. find
  13. Zekun Li, Wenhu Chen, Shiyang Li, Hong Wang, Jing Qian, and Xifeng Yan. 2022. Controllable dialogue simulation with in-context learning. arXiv preprint arXiv:2210.04185. find
  14. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In ACL, pages 3214–3252. Association for Computational Linguistics. find our card
  15. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. arXiv preprint arXiv:2103.10385. find our card
  16. OpenAI. 2023a. Chatgpt: Optimizing language models for dialogue. find
  17. OpenAI. 2023b. Gpt-4 technical report. find our card
  18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  19. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonatha… find our card
  20. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  21. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford\_alpaca. ↗
  22. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239. find our card
  23. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv pr… find
  24. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. find
  25. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In ICLR. OpenReview.net. find
  26. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In ACL, pages 1–9. Association for Computational Linguistics. find our card
  27. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In ACL, pages 4791–4800. Association for Computational Linguistics. find
  28. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536. find our card
  29. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find