Elamur Research⌕
☰
home / papers / 2024 / LLaMA Beyond English: An Empirical Study on Language Capability Transfer

LLaMA Beyond English: An Empirical Study on Language Capability Transfer

Paper · 2024 · Jun Zhao, Zhihao Zhang, Luhui Gao et al.

Summary

Most widely used open LLMs, LLaMA among them, learn mainly from English text and perform poorly in other languages. This study asks which adaptation steps are actually needed to carry LLaMA's text generation and instruction-following abilities into a new language, and how much training each step requires. Using Chinese as the main test case, the authors spent over 1440 GPU hours comparing LLaMA, LLaMA2 and public Chinese variants that had received 30B to 100B extra tokens. They varied three factors: vocabulary extension, further pretraining scale and instruction tuning size. Knowledge was measured with C-Eval, MMLU, AGI-Eval and GAOKAO-Bench. Answer quality was judged on LLM-Eval for accuracy, fluency, informativeness, coherence and harmlessness. Extending the vocabulary proved unnecessary and was even counterproductive at modest training scales. A model trained on less than 1% of the usual further-pretraining data matched the state-of-the-art Open Chinese LLaMA. The same pattern appeared across 13 low-resource languages. For teams building models for under-resourced languages, the results point to a low-cost recipe: keep the original tokenizer, invest in instruction data, and train on several languages together to preserve English skills.

Key points

  • Vocabulary extension did not help. Further pretraining on just 0.5B Chinese tokens with the original vocabulary beat an extended-vocabulary model (Chinese LLaMA, 49,953-token SentencePiece vocabulary) trained on over 30B tokens.
  • Knowledge gains need very large scale. Up to roughly 100B tokens of Chinese pretraining barely raised LLaMA's knowledge scores.
  • Response quality is cheap to improve. Hundreds of thousands of instruction examples were enough, without large-scale pretraining.
  • English skills are at risk. Transfer training on Chinese alone clearly degraded LLaMA's English ability, and multilingual joint training largely prevented this loss.
  • Setup: the models compared were LLaMA-7B/13B, LLaMA2-7B, Chinese LLaMA 7B Plus, Chinese LLaMA2 7B and Open Chinese LLaMA. LLM-Eval covers 17 instruction categories. The authors also report code-switching in model outputs (roughly 1.6–4.8% across languages) and read it as a sign of internal cross-lingual alignment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui and Xuanjing Huang. 2024. LLaMA Beyond English: An Empirical Study on Language Capability Transfer. arXiv:2401.01055.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.01055 · arXiv 2401.01055

@misc{zhao2024llama,
  title = {{LLaMA Beyond English: An Empirical Study on Language Capability Transfer}},
  author = {Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui and Xuanjing Huang},
  year = {2024},
  doi = {10.48550/arxiv.2401.01055},
  eprint = {2401.01055},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.01055}
}

References

58 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anil, R.; Dai, A. M.; Firat, O.; Johnson, M.; and Lepikhin, D. 2023. PaLM 2 Technical Report. arXiv:2305.10403.
  2. Artetxe, M.; Ruder, S.; and Yogatama, D. 2020. On the Cross-lingual Transferability of Monolingual Representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4623–4637.
  3. Online: Association for Computational Linguistics. Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; Nori, H.; Palangi, H.; Ribeiro, M. T.; and Zhang, Y. 2023. find
  4. Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv:2303.12712. Cahyawijaya, S.; Lovenia, H.; Aji, A. F.; Winata, G. I.; and Wilie, B. 2023. NusaCrowd: Open Source Initiative for Indonesian NLP Resources. arXiv:2212.09648.
  5. Cao, S.; Kitaev, N.; and Klein, D. 2020. Multilingual Alignment of Contextual Word Representations. arXiv:2002.03518. find
  6. Chen, G.; Ma, S.; Chen, Y.; Zhang, D.; Pan, J.; Wang, W.; and Wei, F. 2022. Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation. arXiv:2110.08547. — [arXiv:2110.08547](https://arxiv.org/abs/2110.08547) find
  7. Chi, E. A.; Hewitt, J.; and Manning, C. D. 2020. find
  8. Finding Universal Grammatical Relations in Multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5564–5577. Online: Association for Computational Linguistics. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulm… — Training Verifiers to Solve Math Word Problems
  9. Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; and Stoyanov, V. 2020a. Unsupervised Cross-lingual Representation Learning at Scale. arXiv:1911.02116.
  10. Conneau, A.; Wu, S.; Li, H.; Zettlemoyer, L.; and Stoyanov, V. 2020b. Emerging Cross-lingual Structure in Pretrained Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6022– 6034. Online: Association for Computational Linguistics.
  11. Conover, M.; Hayes, M.; Mathur, A.; Xie, J.; Wan, J.; Shah, S.; Ghodsi, A.; Wendell, P.; Zaharia, M.; and Xin, R. 2023. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM. find
  12. Cui, Y.; Yang, Z.; and Yao, X. 2023a. Chinese LLaMA and Alpaca Large Language Models. find
  13. Cui, Y.; Yang, Z.; and Yao, X. 2023b. Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca. arXiv:2304.08177.
  14. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. find
  15. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. Minneapolis, Minnesota: As…
  16. Dong, Q.; Li, L.; Dai, D.; Zheng, C.; Wu, Z.; Chang, B.; Sun, X.; Xu, J.; Li, L.; and Sui, Z. 2023. A Survey on In-context Learning. arXiv:2301.00234.
  17. Dufter, P.; and Schütze, H. 2020. Identifying Elements Essential for BERT’s Multilinguality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4423–4437. Online: Association for Computational Linguistics.
  18. Eronen, J.; Ptaszynski, M.; and Masui, F. 2023. Zero-shot cross-lingual transfer language selection using linguistic similarity. Information Processing & Management, 60(3): 103250. find
  19. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. Measuring Massive Multitask Language Understanding. CoRR, abs/2009.03300.
  20. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. CoRR, abs/2106.09685.
  21. Huang, H.; Tang, T.; Zhang, D.; Zhao, W. X.; Song, T.; Xia, Y.; and Wei, F. 2023a. Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting. arXiv:2305.07004.
  22. Huang, W.; Abbeel, P.; Pathak, D.; and Mordatch, I. 2022. find
  23. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvari, C.; Niu, G.; and Sabato, S., eds., Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Rese…
  24. Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; and Zhang, J. 2023b. find
  25. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. arXiv:2305.08322. Ji, Y.; Deng, Y.; Gong, Y.; Peng, Y.; Niu, Q.; Ma, B.; and Li, X. 2023. BELLE: Be Everyone’s Large Language model Engine. https://github.com/LianjiaTech/BELLE. ↗
  26. Jiang, Z.; Anastasopoulos, A.; Araki, J.; Ding, H.; and Neubig, G. 2020. X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 5943–5959. find
  27. Online: Association for Computational Linguistics. Joshi, P.; Santy, S.; Budhiraja, A.; Bali, K.; and Choudhury, M. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6282–6293.…
  28. Katz, D. M.; Bommarito, M. J.; Gao, S.; and Arredondo, P. 2023. Gpt-4 passes the bar exam. Available at SSRN 4389233. find
  29. Khanuja, S.; Dandapat, S.; Srinivasan, A.; Sitaram, S.; and Choudhury, M. 2020. GLUECoS: An Evaluation Benchmark for Code-Switched NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 3575–3585. Online: Association for Computational Linguistics.
  30. Krishnan, J.; Anastasopoulos, A.; Purohit, H.; and Rangwala, H. 2021. Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Filling. arXiv:2103.07792. find
  31. Li, H.; Koto, F.; Wu, M.; Aji, A. F.; and Baldwin, T. 2023. Bactrian-X : A Multilingual Replicable Instruction-Following Model with Low-Rank Adaptation. arXiv:2305.15011.
  32. Lin, X. V.; Mihaylov, T.; Artetxe, M.; Wang, T.; Chen, S.; Simig, D.; Ott, M.; Goyal, N.; Bhosale, S.; Du, J.; Pasunuru, R.; Shleifer, S.; Koura, P. S.; Chaudhary, V.; O’Horo, B.; Wang, J.; Zettlemoyer, L.; Kozareva, Z.; Diab, M.; Stoyanov, V.; and Li, X. 2022. Few-shot Learning with Multilingual La…
  33. Nguyen, X.-P.; Aljunied, S. M.; Joty, S.; and Bing, L. 2023. Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts. arXiv:2306.11372.
  34. OpenAI. 2022. Introducing ChatGPT. find
  35. OpenLMLab. 2023. Open-Chinese-LLaMA. find
  36. Penedo, G.; Malartic, Q.; Hesslow, D.; Cojocaru, R.; Cappelli, A.; Alobeidli, H.; Pannier, B.; Almazrouei, E.; and Launay, J. 2023. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. arXiv:2306.01116. find
  37. Pires, T.; Schlinger, E.; and Garrette, D. 2019. How Multilingual is Multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4996–5001. Florence, Italy: Association for Computational Linguistics.
  38. Ranta, A.; and Goutte, C. 2021. Linguistic Diversity in Natural Language Processing. Traitement Automatique des Langues, 62(3): 7–11. find
  39. Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; Ilić, S.; Hesslow, D.; and Castagné, R. 2023. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv:2211.05100. find
  40. StabilityAI. 2023. Announcing StableCode. find
  41. Tan, S.; and Joty, S. 2021. Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots. arXiv:2103.09593. find
  42. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Alpaca: A Strong, Replicable Instruction-Following Model. find
  43. Team, I. 2023a. Internlm: A multilingual language model with progressively enhanced capabilities. find
  44. Team, I. 2023b. InternLM: A Multilingual Language Model with Progressively Enhanced Capabilities. https://github.com/InternLM/InternLM-techreport. ↗
  45. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023a. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
  46. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; and Almahairi, A. 2023b. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288.
  47. Winata, G. I.; Cahyawijaya, S.; Liu, Z.; Lin, Z.; Madotto, A.; and Fung, P. 2021a. Are Multilingual Models Effective in Code-Switching? arXiv:2103.13309. find
  48. Winata, G. I.; Lin, Z.; and Fung, P. 2019. Learning Multilingual Meta-Embeddings for Code-Switching Named Entity Recognition. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), 181–186. Florence, Italy: Association for Computational Linguistics. find
  49. Winata, G. I.; Lin, Z.; Shin, J.; Liu, Z.; and Fung, P. 2019. find
  50. Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3541–3547. Hong Kong, China: Association for C… find
  51. Winata, G. I.; Madotto, A.; Lin, Z.; Liu, R.; Yosinski, J.; and Fung, P. 2021b. Language Models are Few-shot Multilingual Learners. In Proceedings of the 1st Workshop on Multilingual Representation Learning, 1–15. Punta Cana, Dominican Republic: Association for Computational Linguistics.
  52. Wu, S.; and Dredze, M. 2019. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT.
  53. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 833–844. Hong Kong, China: Association for Computational Linguistics. find
  54. Wu, Z.; Papadimitriou, I.; and Tamkin, A. 2022. Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies. arXiv:2202.12312. — [arXiv:2202.12312](https://arxiv.org/abs/2202.12312) find
  55. Zhang, M.; Zhang, Q.; Zhang, Y.; and Gui, T. 2023a. find
  56. LLMEVAL-1 Chinese Large Language Model Evaluation Phase 1. Zhang, X.; Li, C.; Zong, Y.; Ying, Z.; He, L.; and Qiu, X. 2023b. Evaluating the Performance of Large Language Models on GAOKAO Benchmark. arXiv:2305.12474.
  57. Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2023. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. arXiv:2304.06364.
  58. Zhu, W.; Liu, H.; Dong, Q.; Xu, J.; Huang, S.; Kong, L.; Chen, J.; and Li, L. 2023. Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis. arXiv:2304.04675.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2024-llama-english-empirical-study-language/