Elamur Research⌕
☰
home / papers / 2024 / Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabic

Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabic

Summary

Most Arabic-focused LLMs have at least 7B parameters, which makes them costly to run and slow at inference. The authors show that a much smaller model can compete. They took Stable LM 2 1.6B, a multilingual model with weak Arabic generation, and continued pre-training it on a mix of English and cleaned Arabic text: web data from CulturaX, SANAD news articles and a Hindawi e-book corpus. The Arabic data went through a datatrove cleaning pipeline before training. For the chat variant, they added a large synthetic Arabic dialogue set produced by having an LLM rephrase existing material. The resulting ar-stablelm-2-base and ar-stablelm-2-chat models match or beat models with up to eight times more parameters on cultural-alignment and understanding benchmarks such as ArabicMMLU in cloze format. The work offers a practical recipe for adapting a small model to a new language under a fixed tokenizer, and it shows that synthetic instruction data can meaningfully help.

Key points

  • After cleaning, the Arabic data totalled about 114B tokens. Training ran for 500k steps, consumed roughly 197B tokens, and used two nodes with 8 H100s each, at about 400K tokens per batch.
  • Tokenizer fertility (sub-word tokens per word) was measured on OSCAR and the PADT treebank:
    • Jais has the lowest fertility.
    • AceGPT, built on Llama 2's 32k vocabulary, has the highest.
    • Stable LM 2's Arcade100k tokenizer (vocabulary 100,289) is second highest.
  • Because the inherited tokenizer splits Arabic so heavily, Arabic was over-sampled about 4.6× relative to its raw share. The final English:Arabic sampling ratio was around 4.6:1, against a raw token ratio of roughly 5:1.
  • Cleaning combined several filters:
    • unsafe-phrase lists (about 400 phrases; 3 or more hits removes a document);
    • ad-phrase lists;
    • line-count rules;
    • Gopher heuristics with statistics recomputed for Arabic;
    • Unicode normalization of Arabic character variants;
    • regex removal of leading titles and dates.
  • In one comparison table, the chat model averages 49.6 against 47.2 for jais-13b. An earlier learning-rate cool-down (starting near 120B tokens) beat a late one (near 190B) on ArabicMMLU.

Interesting ideas

  • I1. Fertility was measured on two kinds of text, a noisy web corpus (OSCAR) and a carefully segmented treebank (PADT), giving a cheap view of how badly a tokenizer fragments words. — For Mova (tokens): Compute split rates of the en tokenizer on both UD English treebanks and raw web text, so that mismatches between them flag segmentation rules that drift from UD conventions.
  • I2. Instead of mixing sources by raw token counts, the authors adjusted sampling to offset a known measurement bias (high fertility inflating Arabic token counts). — For Mova (learn): When mixing silver data from different witness parsers or genres, weight sources by sentences or meaning units rather than tokens, so verbose sources do not dominate training.
  • I3. Generic quality filters like Gopher were not reused as-is; their thresholds were re-derived from statistics of the target language. — For Mova (learn): Filter silver trees with thresholds recalculated per genre (fables, school texts, word problems) rather than one global cutoff, keeping valid but unusual sentences.
  • I4. The same benchmark was scored in both cloze and multiple-choice formats, and the per-model difference was plotted, exposing how much format alone shifts results. — For Mova (answer): Run school-course tests both as free answer and as option selection, and treat a large gap as a diagnostic that the reader is exploiting format cues rather than understanding.
  • I5. An ablation showed that where the learning-rate cool-down starts noticeably changes downstream quality, not just the final loss. — For Mova (learn): When training gparse, compare early versus late decay schedules on LAS from held-out treebanks, instead of fixing the schedule by habit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.04277 · arXiv 2412.04277

@misc{alyafeai2024arabic,
  title = {{Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabic}},
  author = {Alyafeai, Zaid and Pieler, Michael and Teufel, Hannah and Tow, Jonathan and Bellagente, Marco and Phung, Duy and Pinnaparaju, Nikhil and Adithyan, Reshinth and Rocha, Paulo and Zhuravinskyi, Maksym and Riquelme, Carlos},
  year = {2024},
  doi = {10.48550/arxiv.2412.04277},
  eprint = {2412.04277},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.04277}
}

References

44 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yi Zheng, Mehdi Rezagholizadeh, Baoxing Huai, et al. 2023. Aramus: Pushing the limits of data and model scale for arabic natural language processing. arXiv preprint arXiv:2306.06800. ↗
  2. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. AlGhafa evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023, pages 244–275, Singapore (Hybrid). Association for Computational Linguistics. find
  3. Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran A. Q. Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, Yousef Ali, and Maged S. Al-Shaibani. 2024. CIDAR: Culturally Relevant Instruction Dataset For Arabic. Preprint, arXiv:2402.03177. ↗
  4. Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, et al. 2024. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. arXiv preprint arXiv:2402.01781. ↗
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Aragpt2: Pre-trained transformer for arabic language generation. arXiv preprint arXiv:2012.15520. ↗
  6. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. ↗ our card
  7. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. ↗ our card
  8. Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. 2024. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834. ↗
  9. Hasna Chouikhi, Manel Aloui, Cyrine Ben Hammou, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. 2024. Llamar & gemmar: Enhancing llms through arabic instruction-tuning. arXiv preprint arXiv:2407.02147. ↗ our card
  10. Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. arXiv eprints, arXiv:2201.06642. ↗ our card
  11. Omar Einea, Ashraf Elnagar, and Ridhwan Al Debsi. 2019. Sanad: Single-label arabic news articles dataset for automatic text categorization. Data in brief, 25:104076. find
  12. Asaad Alghamdi, Xinyu Duan, Wei Jiang, Zhenhai Wang, Yimeng Wu, Qingrong Xia, Zhefeng Wang, Leo Gao. 2021. Multiple choice normalization in lm evaluation. find
  13. Abbas Ghaddar, Yimeng Wu, Sunyam Bagga, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Duan Xinyu, Zhefeng Wang, et al. 2022. Revisiting pre-trained language models and their evaluation for arabic natural language understanding. arXiv preprint arXiv:2205.10687. ↗ our card
  14. Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. 2024. Changing answer order can decrease mmlu accuracy. arXiv preprint arXiv:2406.19470. ↗
  15. Jan Hajic, Otakar Smrz, Petr Zemánek, Jan Šnaidauf, and Emanuel Beška. 2004. Prague arabic dependency treebank: Development in data and tools. In Proc. of the NEMLAR Intern. Conf. on Arabic Language Resources and Tools, volume 1. find
  16. A. Hallberg. 2024. The arabic e-book corpus (version 1) [data set]. DOI: Available after publication. find
  17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗ our card
  18. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al. 2023. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053. ↗ our card
  19. Inception. 2024. Jais family model card. find
  20. "Fajri Koto, Haonan Li, Sara Shatanawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin". "2024". "arabicmmlu: Assessing massive multitask language understanding in arabic". Preprint, arXiv:"2402.12840". find our card
  21. Imad Lakim, Ebtesam Almazrouei, Ibrahim Abualhaol, Merouane Debbah, and Julien Launay. 2022. A holistic assessment of the carbon footprint of noor, a very large arabic language model. In Proceedings of Big-Science Episode# 5–Workshop on Challenges & Perspectives in Creating Large Language Models, pages 84–94. find
  22. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. 2024. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794. ↗ our card
  23. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. ↗
  24. Tiedong Liu and Bryan Kian Hsiang Low. 2023. Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks. arXiv preprint arXiv:2305.14201. ↗
  25. Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. 2024. Quantifying variance in evaluation benchmarks. arXiv preprint arXiv:2406.10229. ↗
  26. Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380. ↗
  27. Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulić. 2024. Zero-shot tokenizer transfer. arXiv preprint arXiv:2405.07883. ↗
  28. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2021. Arat5: Text-to-text transformers for arabic language generation. arXiv preprint arXiv:2109.12068. ↗ our card
  29. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. arXiv preprint arXiv:2309.09400. ↗ our card
  30. Michael Pieler, Marco Bellagente, Hannah Teufel, Duy Phung, Nathan Cooper, Jonathan Tow, Paulo Rocha, Reshinth Adithyan, Zaid Alyafeai, Nikhil Pinnaparaju, Maksym Zhuravinskyi, and Carlos Riquelme. 2024. Rephrasing natural text data with different languages and quality levels for large language model pre-training. Preprint, arXiv:2410.20796. ↗
  31. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  32. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. ↗
  33. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints. find our card
  34. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. find our card
  35. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗ our card
  36. Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. 2024. Aya dataset: An open-access collection for multilingual instruction tuning. Preprint, arXiv:2402.06619. ↗ our card
  37. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. find
  38. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. ↗
  39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  40. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. 2024. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827. ↗ our card
  41. Haochun Wang, Sendong Zhao, Zewen Qiang, Bing Qin, and Ting Liu. 2024. Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models. arXiv preprint arXiv:2402.01349. ↗
  42. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. ↗
  43. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36. Jianqing Zhu, Huang Huang, Zhihang Lin, Juhao Liang, Zhengyang Tang, Khalid Almubarak, Abdulmohsen Alharthi, Bang An, Juncai He, Xiangbo Wu, et al. Second language (arabic) acquisition of llms via progressive vocabulary expansion. find
  44. Walelign Tewabe Sewunetie, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Hellina Hailu Nigatu, Gashaw Kidanu, Zewdie Mossie, Hussien Seid, Eshete Derb, and Seid Muhie Yimam. 2024. Evaluating gender bias in machine translation for low-resource languages. In 5th Workshop on African Natural Language Processing. *Figure 5: ArabicMMLU benchmark results of the Arabic Stable LM 1.6B base model with the cloze format (CF) and multiple choice format (MCF).* Benchmarks like ArabicMMLU (Koto et al., "2024") evaluate LLMs on natural language understanding. The multiple-choice data can be used to evaluate the models with two different setups: 1. Multiple choice format (MCF) The multiple choice format asks the model to pick the letter associated with the correct answer. For example: What is the capital of Spain? A. Paris B. London C. Madrid. with the correct answer C. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alyafeai-2024-arabic-stable-lm-adapting-stable/