Elamur Research
☰
home / papers / 2023 / Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback

Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.emnlp-demo.28.pdf

Summary

Open-source instruction-following LLMs have mostly been tuned for English and a handful of major languages. Earlier multilingual attempts, such as Phoenix and Bactrian-X, used only supervised fine-tuning (SFT). Okapi is presented as the first framework to apply RLHF to open-source LLMs across many languages.

The authors grew the Alpaca instruction set with Self-Instruct and then had ChatGPT translate everything into 26 languages: instructions, inputs and outputs. These languages range from well-resourced ones to poorly resourced ones such as Telugu or Kannada. ChatGPT was also prompted to rank several candidate responses, and these rankings were used to train reward models. For evaluation, they translated three benchmarks from the HuggingFace Open LLM Leaderboard into the same languages.

Using BLOOM and LLaMa (7B) as base models, RLHF-tuned versions generally outperformed SFT-tuned ones on most datasets and language groups. Gains were smallest for low-resource languages. The released data, models and scripts give a reusable starting point for multilingual alignment research, including Ukrainian.

Key points

  • English instruction data: 106K new English instructions were generated with Self-Instruct, starting from 175 seed tasks and using text-davinci-003. A ROUGE-based filter kept them distinct from Alpaca's 52K, giving 158K instructions in total.
  • Languages: the 26 target languages are grouped by CommonCrawl share into 8 high-, 11 medium- and 7 low-resource languages. Ukrainian (uk) falls in the medium group.
  • Evaluation benchmarks: ARC, HellaSwag and MMLU were machine-translated into all 26 languages to evaluate generative models.
  • Comparison with BLOOMZ: BLOOM 7B tuned on Okapi's 158K instructions (via SFT or RLHF) beat BLOOMZ on nearly all languages. BLOOMZ had been trained on over 78M template-based multilingual instructions, so the authors read this as evidence that their data is of higher quality.
  • Release and limitations: resources are available at github.com/nlp-uoregon/Okapi, and the models are limited to research use. Because the data comes from GPT-3 and ChatGPT, the models may inherit their hallucination, bias and toxicity problems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

167 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 160 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-demo.28

@inproceedings{lai2023okapi,
  title = {{Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback}},
  author = {Lai, Viet and Nguyen, Chien and Ngo, Nghia and Nguyen, Thuat and Dernoncourt, Franck and Rossi, Ryan and Nguyen, Thien},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-demo.28},
  url = {https://doi.org/10.18653/v1/2023.emnlp-demo.28}
}

References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang, Juhao Liang, Chen Zhang, Zhiyi Zhang, Jianquan Li, Xiang Wan, Benyou Wang, and Haizhou Li. 2023. Phoenix: Democratizing chatgpt across languages. ArXiv, abs/2304.10453. find
  2. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  3. Jonathan Choi, Kristin Hickman, Amy Monahan, and Daniel Schwarcz. 2023. Chatgpt goes to law school. Available at SSRN. find
  4. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS). find
  5. Hyung Won Chung, Le Hou, S. Longpre, and Barret Zoph et al. 2022. Scaling instruction-finetuned language models. ArXiv, abs/2210.11416. find
  6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457. find
  7. Together Computer. 2023. Redpajama: An open source recipe to reproduce llama training dataset. find
  8. Mike Conover, Matt Hayes, and Ankit Mathur et al. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. https://www. databricks.com. find
  9. Leo Gao, Jonathan Tow, Stella Biderman, and et al. 2021. A framework for few-shot language model evaluation. find
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. find our card
  11. HuggingFace. 2023. Open llm leaderboard. https: //github.com/tatsu-lab/stanford\_alpaca. find
  12. Ebtesam Almazrouei, Hamza Alobeidli, and Abdulaziz Alshamsi et al. 2023. Falcon-40B: an open large language model with state-of-the-art performance. find
  13. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. ArXiv, abs/2302.04… find our card
  14. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine. ArXiv, 2301.08745. find our card
  15. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of language agents. ArXiv, abs/2103.14659. find
  16. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning. ArXiv, abs/2304.05613. find our card
  17. Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-x: A multilingual replicable instruction-following model with low-rank adaptation. ArXiv, abs/2305.15011. find
  18. S. Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. ArXiv, abs/2301.13688. find our card
  19. Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Survey. find our card
  20. MosaicML. 2023. Introducing mpt-7b: A new standard for open-source, commercially usable llms. https: //www.mosaicml.com/blog/mpt-7b. find
  21. Niklas Muennighoff, Thomas Wang, and Lintang Sutawika et al. 2022. Crosslingual generalization through multitask finetuning. ArXiv, abs/2211.01786. find our card
  22. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan Rossi, and Thien Huu Nguyen. 2023. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. ArXiv, abs/2309.09400. find our card
  23. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… find
  24. Jack Rae, Sebastian Borgeaud, and et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. ArXiv, abs/2112.11446. find
  25. Victor Sanh, Albert Webson, and Colin Raffel et al. 2021. Multitask prompted training enables zero-shot task generalization. ArXiv, abs/2110.08207. find our card
  26. Teven Scao, Angela Fan, and et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. ArXiv, abs/2211.05100. find
  27. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. ArXiv, abs/2009.01325. find
  28. Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. 2021. Understanding the capabilities, limitations, and societal impact of large language models. ArXiv, abs/2102.02503. find
  29. Rohan Taori, Ishaan Gulrajani, and Tianyi Zhang et al. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford\_alpaca. ↗
  30. Hugo Touvron, Thibaut Lavril, and Gautier Izacard et al. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971. find
  31. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  32. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. In Proceedings of the International Conference on Learning Representations. find
  33. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transaction… find our card
  34. Laura Weidinger, John F. J. Mellor, and Maribeth Rauh et al. 2021. Ethical and social risks of harm from language models. ArXiv, abs/2112.04359. find our card
  35. Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. 2023. Lamini-lm: A diverse herd of distilled models from large-scale instructions. ArXiv, abs/2304.14402. find
  36. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. find
  37. StabilityAI. 2023. Stablelm: Stability ai language models. https://github.com/stability-AI/ stableLM. ↗