Elamur Research⌕
☰
home / papers / 2023 / Phoenix: Democratizing ChatGPT across Languages

Phoenix: Democratizing ChatGPT across Languages

Paper · 2023 · Zhihong Chen, Feng Jiang, Junying Chen et al.

Summary

Open ChatGPT-style assistants released in early 2023 were mostly built for English, or English plus Chinese. Speakers of other languages, especially those using non-Latin scripts, had few usable options. The authors present Phoenix, a 7B model post-trained from the multilingual BLOOMZ backbone. Its recipe combines two kinds of supervision that Alpaca and Vicuna had used separately: instruction data and user-shared ChatGPT conversations. Instructions were made multilingual in two ways. In one, the finished answer is translated, which preserves answer quality. In the other, the prompt is translated and then answered, which allows culture-specific responses. Evaluation uses 100 questions across ten categories, scored by GPT-4 and by human volunteers. Phoenix leads open-source Chinese models and clearly beats Guanaco in Arabic, Japanese and Korean. It trails English-focused models, a cost the authors call the "multilingual tax". A LLaMA-based variant, Chimera, avoids that tax for Latin-script languages. The work is an early, fully released multilingual assistant, with code, data and weights in LLMZoo. However, its evaluation is small and, by the authors' own admission, not rigorous. Ukrainian is not evaluated separately.

Key points

  • Models: Phoenix uses BLOOMZ-7B. Chimera uses LLaMA 7B/13B. Table 1 lists about 267K instructions and 189K conversations covering 40+ languages, though the extracted table is garbled. Everything is released via GitHub (FreedomIntelligence/LLMZoo).
  • Chinese results: GPT-4 rated Phoenix at 122.7% of BELLE-7B-2m, 135.3% of Chinese-Alpaca-7b and 125.2% of Chinese-Alpaca-13b. Human evaluation shows win margins above 50% over the open Chinese baselines.
  • English results: Phoenix scores about 21% above Vicuna-7b and ChatGLM-6b but stays behind Vicuna-13b and ChatGPT. GPT-4 rated Chimera at 96.6% "ChatGPT Quality", which the authors claim as a new open-source SOTA.
  • Ablation: Adding instruction data to conversation data improved results by 5.1% relative in Chinese and 6.3% in English, measured against GPT-3.5 Turbo.
  • Limitations: Evaluation relies on only 100 questions and mostly on GPT-4 judging. Human checks covered Chinese only. The authors promise more rigorous evaluation later.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong KongHong Kong · China53 works · 12,921 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China17 works · 2,364 citations🎓Shenzhen Research Institute of Big DataShenzhen · China11 works · 1,615 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 467 works

Cite

Cite as Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang, Juhao Liang, Chen Zhang, Zhiyi Zhang, Jianquan Li, Xiang Wan, Benyou Wang and Haizhou Li. 2023. Phoenix: Democratizing ChatGPT across Languages. arXiv:2304.10453.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.10453 · arXiv 2304.10453

@misc{chen2023phoenix,
  title = {{Phoenix: Democratizing ChatGPT across Languages}},
  author = {Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang, Juhao Liang, Chen Zhang, Zhiyi Zhang, Jianquan Li, Xiang Wan, Benyou Wang and Haizhou Li},
  year = {2023},
  doi = {10.48550/arxiv.2304.10453},
  eprint = {2304.10453},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.10453}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asma Ben Abacha, Sadid A. Hasan, Vivek Datla, Joey Liu, Dina Demner-Fushman, and Henning Müller. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In CLEF, 2019. 2 find
  2. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 2, 3, 4, 5, 7, 9 find
  3. Yiming Cui and Ziqing Yang. Chinese llama and alpaca llms. https://github.com/ymcui/ Chinese-LLaMA-Alpaca, 2023. 4, 5 ↗
  4. Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. 2, 3 find
  5. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection, 2023. 5
  6. Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases. arXiv preprint arXiv:2303.14742, 2023. 2, 4
  7. Ziang Leng, Qiyuan Chen, and Cheng Li. Luotuo: An instruction-following chinese language model, lora tuning on llama. https://github.com/LC1332/Chinese-alpaca-lora, 2023. 4 ↗
  8. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  9. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. 2
  10. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. 4, 6
  11. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. 4
  12. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. 4, 6 ↗
  13. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. 2, 4
  14. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022. 4, 5 find
  15. BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît… find
  16. Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data, 2023. 4, 5
  17. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh Inter…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-phoenix-democratizing-chatgpt-across-languages/