Elamur Research⌕
☰
home / papers / 2025 / Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion

Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion

Paper · 2025 · Jianqing Zhu, Huang Huang, Zhihang Lin et al.

Summary

English-centric LLMs such as LLaMA2 handle Arabic poorly at the vocabulary level. Their tokenizer holds little beyond basic Arabic characters, so Arabic text splits into near-letter sequences and generation is slow. Adding many Arabic subwords at once is risky: suddenly the model sees many units it has no trained representation for, and this can erode what it already knows.

The authors draw an analogy with how humans gain second-language vocabulary bit by bit. They propose Progressive Vocabulary Expansion, built on Incremental BPE (I-BPE). The vocabulary grows in stages. After each growth step, training continues on data with a rising share of text covered by the new tokens, which keeps the OOV ratio moderate throughout. They compare uniform and exponential growth schedules.

Applied on top of LLaMA2, the method yields AraLLaMA (7B and 13B, base and chat). A TinyLLaMA ablation shows the staged method beats one-shot expansion. The resulting model produces Arabic words several times faster and matches strong Arabic LLMs while largely keeping its English ability. Weights and data pipeline are released.

Key points

  • I-BPE takes three inputs: a starting vocabulary, a target vocabulary size for each stage, and a corpus share for each stage's new tokens. The model is trained at each stage before the vocabulary grows again.

  • Two schedules were studied: adding a constant number of tokens per stage, or doubling the additions each stage.

  • Tokenizer quality on Arabic:

    • The compression ratio is 0.3174, reported as 68% better than LLaMA2.
    • Subword fertility is 1.7063, against 5.3844 for LLaMA2, 5.2833 for Mistral and 2.0668 for Bloomz.

  • TinyLLaMA ablation compared a baseline, one-shot vocabulary expansion (VE) and the progressive method (PVE):

    BenchmarkBaselineVEPVE
    ArabicMMLU average36.538.540.7
    Arabic Vicuna-8021.30%22.61%29.18%

  • Generation speed versus LLaMA2 7B chat:

    • Speed in tokens per second is about the same, around 30 for both models.
    • Arabic output rises from 4.55 to 20.37 words per second, roughly 4.5×.
    • On English MMLU, AraLLaMA base scores slightly below LLaMA but above AceGPT.
    • The model has not yet been evaluated by native Arabic speakers.

Interesting ideas

  • I1. Grow the inventory of units in stages rather than all at once, so that only a small fraction of input is unfamiliar at any moment and earlier skills survive.
  • I2. Tie each expansion step to a data-mix knob, so the share of training text that exercises the new material ramps up gradually instead of jumping.
  • I3. Measure speed in user-facing units (words per second), not internal units (tokens per second). Equal token throughput can hide a large real difference.
  • I4. Validate the idea on a small model with three arms (no change, naive change, proposed change) before scaling up. Pair this with a check that the original language's ability did not drop.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): When the en crate's lexicon or morphology gains a large batch of new domain lemmas or multiword units, we could admit them in waves and check tagger accuracy after each wave, so we can locate any wave that degrades it.
  • I2 (learn): When retraining the perceptron parser or gparse on a new treebank or converted UD dialect, we could ramp the share of new-source sentences per epoch rather than mixing everything at once.
  • I3 (back): Generation and the text → graph → text compressor could report cost per output word or per graph node, so tokenizer or representation changes are judged on what users actually get.
  • I4 (learn): Any new repair-rule or silver-data scheme could be tested this way, with a naive-variant control and a check that LAS on the original treebank holds.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🎓King Abdullah University of Science and TechnologyJeddah · Saudi Arabia27 works · 1,084 citations🎓Shenzhen Research Institute of Big DataShenzhen · China17 works · 1,721 citations🎓King Abdulaziz UniversityJeddah · Saudi Arabia31 works · 508 citations🎓Prince Sattam Bin Abdulaziz UniversityAl Kharj · Saudi Arabia17 works · 489 citations🎓University of BishaBisha · Saudi Arabia4 works · 56 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Jianqing Zhu, Huang Huang, Zhihang Lin, Juhao Liang, Zhengyang Tang, Khalid Almubarak, Mosen Alharthi, Bang An, Juncai He, Xiangbo Wu, Fei Yu, Junying Chen, Ma Zhuoheng, Yuhao Du, He Zhang, Saied Alshahrani, Emad A. Alghamdi, Lian Zhang, Ruoyu Sun, Haizhou Li, Benyou Wang and Jinchao Xu. 2025. Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.100

@inproceedings{zhu2025second,
  title = {{Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion}},
  author = {Jianqing Zhu, Huang Huang, Zhihang Lin, Juhao Liang, Zhengyang Tang, Khalid Almubarak, Mosen Alharthi, Bang An, Juncai He, Xiangbo Wu, Fei Yu, Junying Chen, Ma Zhuoheng, Yuhao Du, He Zhang, Saied Alshahrani, Emad A. Alghamdi, Lian Zhang, Ruoyu Sun, Haizhou Li, Benyou Wang and Jinchao Xu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.100},
  url = {https://doi.org/10.18653/v1/2025.acl-long.100}
}

References

46 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  2. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. Llemma: An open language model for mathematics. Preprint, arXiv:2310.10631. ↗
  3. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. arXiv preprint arXiv:2004.03720. ↗
  4. Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/ codealpaca. ↗
  5. Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang, Juhao Liang, Chen Zhang, Zhiyi Zhang, et al. 2023. Phoenix: Democratizing chatgpt across languages. arXiv preprint arXiv:2304.10453. ↗
  6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  7. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL. ↗
  8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1. ↗
  9. James Coady. 1996. L2 vocabulary acquisition through extensive reading. In Second language vocabulary acquisition: A rationale for pedagogy, pages 225– 237. Cambridge University Press. ↗
  10. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. find
  11. Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177. ↗
  12. Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 find
  13. Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 5427–5444. Association for Computational Linguistics. find
  14. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300. ↗
  15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. ↗
  16. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, et al. 2024. Acegpt, localizing large language models in arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163. ↗
  17. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. ↗
  18. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, et al. 2024. Arabicmmlu: Assessing massive multitask language understanding in arabic. arXiv preprint arXiv:2402.12840. ↗
  19. Stephen Krashen. 1981. Second language acquisition. Second Language Learning, 3(7):19–39. find
  20. Stephen Krashen. 1982. Principles and practice in second language acquisition. Pergamon Press. find
  21. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959. ↗
  22. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794, Copenhagen, Denmark. Association for Computational Linguistics. ↗
  23. Héctor Daniel León Romero et al. 2016. Spaced retrieval practice applied to vocabulary learning in secondary education. find
  24. Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. 2024. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402.13064. ↗
  25. Ibraheem Muhammad Moosa, Mahmud Elahi Akhter, and Ashfia Binte Habib. 2023. Does transliteration help multilingual language modeling? In Findings of the Association for Computational Linguistics: EACL 2023, pages 670–685, Dubrovnik, Croatia. Association for Computational Linguistics. find
  26. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786. ↗
  27. Tatsuya Nakata. 2015. Effects of expanding and equal spacing on second language vocabulary learning: Does gradually increasing spacing increase vocabulary learning? Studies in Second Language Acquisition, 37(4):677–711. ↗
  28. I. S. P. Nation. 2001. Learning Vocabulary in Another Language. Cambridge Applied Linguistics. Cambridge University Press. ↗
  29. Ian SP Nation and I. S. P. Nation. 2001. Learning vocabulary in another language, volume 10. Cambridge University Press. find
  30. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, et al. 2023. Seallms–large language models for southeast asia. arXiv preprint arXiv:2312.00738. ↗
  31. Restrepo Ramos and Falcon Dario. 2015. Incidental vocabulary learning in second language acquisition: A literature review. Profile Issues in TeachersProfes-sional Development, 17(1):157–166. ↗
  32. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135. Association for Computational Linguistics.
  33. Elizabeth Salesky, Andrew Runge, Alex Coda, Jan Niehues, and Graham Neubig. 2020. Optimizing segmentation granularity for neural machine translation. Machine Translation, 34(1):41–59. ↗
  34. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. ↗
  36. Daria Soboleva, Al-Khateeb Faisal, Myers Robert Steeves Jacob R, Hestness Joel, and Dey Nolan. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. find
  37. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford_alpaca. ↗
  38. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. ↗
  39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  40. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. 2024. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827. ↗
  41. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244. ↗
  42. Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. 2020. Vocabulary learning via optimal transport for neural machine translation. arXiv preprint arXiv:2012.15671. ↗
  43. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385. ↗
  44. Songshan Zhang, Hai Xu, and Xian Zhang. 2021. The effects of dictionary use on second language vocabulary acquisition: A meta-analysis. International Journal of Lexicography, 34(1):1–38. ↗
  45. Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055. ↗
  46. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2025-second-language-arabic-acquisition-llms/