Elamur Research⌕
☰
home / papers / 2025 / Kuwain 1.5B: An Arabic SLM via Language Injection

Kuwain 1.5B: An Arabic SLM via Language Injection

Summary

Most open LLMs are trained mainly on English, and they handle Arabic poorly. The usual fix is continued pre-training, which is costly and tends to erode the model's English skills. This paper builds Kuwain 1.5B by "injecting" Arabic into TinyLlama 1.1B, an English-centric base model. The original weights stay frozen. Only newly inserted layers are trained, and the tokenizer gains about 26K Arabic tokens. Training uses the full Arabic dataset plus a small share of the original English data, which keeps the old abilities stable.

The authors report three results compared with the base model:

  • Arabic scores rise by about 8% on average.
  • English holds steady or improves slightly (~1%).
  • Training cost falls by roughly 70%.

On an Arabic leaderboard, the 1.5B model is competitive with systems up to ten times larger. The work matters because it offers a cheap, reusable recipe for adding a language to an existing model without catastrophic forgetting. That makes it relevant to any under-served language, not only Arabic.

Key points

  • Base model: TinyLlama 1.1B. Kuwain grows it to 1.5B by stacking extra layers and trains only those new layers.
  • The vocabulary gains ~26K Arabic tokens. The authors cite prior evidence that English-centric tokenizers can raise multilingual training cost by 68%.
  • Keeping about 20% of the original English data was enough to preserve English while learning Arabic.
  • On the Arabic leaderboard (ARC-c, ARC-e, BoolQ, COPA, HellaSwag, OBQA, PIQA), Kuwain averages ~44.5. That places it next to 7–13B models such as AceGPT 13B and Meta-Llama-3 8B.
  • Related continual pre-training efforts lost English performance: Meltemi-7B −6%, LeoLM-7B −2.6%, Swallow-MS −5.3%. A fine-tuned Kuwain derivative, Lahajawi, is used for Arabic cross-dialect translation.

Interesting ideas

  • I1. Freezing the trained network and adding fresh trainable layers for a new capability puts new knowledge in a separate place, so the old competence is structurally protected rather than merely hoped to survive. — For Mova (tree): gparse could absorb a new UD dialect or text domain through added layers on a frozen base, keeping LAS on the original treebank intact.
  • I2. A small replay share of old-domain data mixed into new-domain training is a cheap guard against forgetting. — For Mova (learn): when retraining on silver data agreed by witness parsers, mix in a fixed fraction of gold treebank sentences so the parser does not drift toward silver-data artifacts.
  • I3. Extending the vocabulary with units native to the new language avoids falling back to character-level pieces, which waste capacity and blur morphology. — For Mova (tokens): when the crate en meets a new register or domain, add explicit lexicon and morphology entries rather than relying on generic subword fallback, keeping lemmas and tags meaningful.
  • I4. Each run is judged on two axes at once, gain on the new target and retention on the old one, so "improvement" cannot hide a regression elsewhere. — For Mova (learn): every training or repair-rule change should report both target-task gains and scores on untouched benchmarks as a negative control.
  • I5. Plotting score against model size (performance per parameter) makes efficiency a first-class result instead of chasing raw accuracy. — For Mova (answer): report accuracy per unit of compute or rule count for the readers, which keeps the explainable pipeline honest against heavier black-box baselines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.15120 · arXiv 2504.15120

@misc{hennara2025kuwain,
  title = {{Kuwain 1.5B: An Arabic SLM via Language Injection}},
  author = {Hennara, Khalil and Chrouf, Sara and Hamed, Mohamed Motaism and Aldallal, Zeina and Hadid, Omar and AlModhayan, Safwan},
  year = {2025},
  doi = {10.48550/arxiv.2504.15120},
  eprint = {2504.15120},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.15120}
}

References

44 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. ↗
  2. Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms. arXiv preprint arXiv:2305.16339, 2023. ↗
  3. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv preprint arXiv:2304.05613, 2023. ↗ our card
  4. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. ↗
  5. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. ↗ our card
  6. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. ↗
  7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. ↗
  8. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024. ↗ our card
  9. M Ali, M Fromm, K Thellmann, R Rutmann, M Lübbering, J Leveling, K Klug, J Ebert, N Doll, JS Buschhoff, et al. Tokenizer choice for llm training: Negligible or crucial? preprint (2023). arXiv preprint arXiv:2310.08754, 2023. ↗
  10. Mohamed Motasim Hamed, Muhammad Hreden, Khalil Hennara, Zeina Aldallal, Sara Chrouf, and Safwan AlModhayan. Lahjawi: Arabic cross-dialect translator. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pages 12–24, 2025. find
  11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1– 113, 2023. find
  12. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. https://inria.hal.science/hal-03850124, 2023. ↗
  13. L Xue. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020. ↗
  14. Terra Blevins and Luke Zettlemoyer. Language contamination helps explain the cross-lingual capabilities of english pretrained models. arXiv preprint arXiv:2204.08110, 2022. ↗
  15. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. ↗
  16. Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. ↗
  17. Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. Multilingual instruction tuning with just a pinch of multilinguality. arXiv preprint arXiv:2401.01854, 2024. ↗
  18. Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training of large language models: How to (re) warm your model? arXiv preprint arXiv:2308.04014, 2023. ↗
  19. Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177, 2023. ↗ our card
  20. Björn Plüster. Leolm: Igniting german-language llm research, 2023. find
  21. Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. Building a large japanese web corpus for large language models. arXiv preprint arXiv:2404.17733, 2024. ↗
  22. Leon Voukoutis, Dimitris Roussis, Georgios Paraskevopoulos, Sokratis Sofianopoulos, Prokopis Prokopidis, Vassilis Papavasileiou, Athanasios Katsamanis, Stelios Piperidis, and Vassilis Katsouros. Meltemi: The first open large language model for greek. arXiv preprint arXiv:2407.20743, 2024. ↗
  23. Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah A Smith, and Luke Zettlemoyer. Breaking the curse of multilinguality with cross-lingual expert language models. arXiv preprint arXiv:2401.10440, 2024. ↗
  24. Hao Zhou, Zhijun Wang, Shujian Huang, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Weihua Luo, and Jiajun Chen. Moe-lpr: Multilingual extension of large language models through mixture-of-experts with language priors routing. arXiv preprint arXiv:2408.11396, 2024. ↗
  25. Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, et al. Bloom+ 1: Adding language support to bloom for zero-shot prompting. arXiv preprint arXiv:2212.09535, 2022. ↗
  26. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149, 2023. ↗ our card
  27. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053, 2023. ↗ our card
  28. Anis Koubaa, Adel Ammar, Lahouari Ghouti, Omar Najar, and Serry Sibaee. Arabiangpt: Native arabic gpt-based large language. arXiv preprint arXiv:2402.15313, 2024. ↗ our card
  29. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan AlRashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. Allam: Large language models for arabic and english. arXiv e-prints, pages arXiv–2407, 2024. find our card
  30. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. arXiv preprint arXiv:2309.09400, 2023. ↗ our card
  31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019. find
  32. Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ping Luo, and Ying Shan. Llama pro: Progressive llama with block expansion. arXiv preprint arXiv:2401.02415, 2024. ↗
  33. Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. ↗
  34. Zoltan Csaki, Pian Pawakapan, Urmish Thakker, and Qiantong Xu. Efficiently adapting pre-trained language models to new languages. arXiv preprint arXiv:2311.05741, 2023. ↗
  35. Aleksandr Nikolich, Konstantin Korolev, and Artem Shelmanov. Vikhr: The family of open-source instruction-tuned large language models for russian. arXiv preprint arXiv:2405.13929, 2024. ↗
  36. Ali Farghaly and Khaled Shaalan. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1–22, 2009. find
  37. Sabri Elkateb, William J Black, Piek Vossen, David Farwell, Horacio Rodriguez, Adam Pease, Musa Alkhalifa, and Christiane Fellbaum. Arabic wordnet and the challenges of arabic. In Proceedings of the International Conference on the Challenge of Arabic for NLP/MT, pages 15–24, 2006. find
  38. Iyad Issa. Morphological complexity in arabic spelling and its implication for cognitive processing. Journal of Psycholinguistic Research, 52(1):331–357, 2023. find
  39. Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arxiv 2018. arXiv preprint arXiv:1808.06226, 1808. ↗ our card
  40. Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887, 2024. ↗
  41. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104, 2020. ↗ our card
  42. Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint arXiv:2407.13623, 2024. ↗
  43. Kaj Bostrom and Greg Durrett. Byte pair encoding is suboptimal for language model pretraining. arXiv preprint arXiv:2004.03720, 2020. ↗
  44. Ali Elfilali, Hamza Alobeidli, Clémentine Fourrier, Basma El Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. Open arabic llm leaderboard. https://huggingfac e.co/spaces/OALL/Open-Arabic-LLM-Leaderboard, 2024. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hennara-2025-kuwain-1-5b-arabic-slm/