AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3

Summary

Tokenizers built mostly on English text break Arabic into too many pieces. Arabic is highly inflected and its spelling varies: there are several Alif forms, optional vowel marks, Arabic-Indic digits and Tatweel. Long token sequences raise compute cost and use up context. The authors build AraToken, a SentencePiece Unigram tokenizer trained on about 120M tokens from Arabic Wikipedia and ArabicText-Large. Before training, text goes through a normalization chain: NFKC, then rules that merge Alif variants, convert digits and punctuation, drop Tatweel and optionally strip diacritics. They compare BPE, WordPiece and Unigram, with and without normalization. Normalized Unigram reaches 1.199 tokens per word, against 1.35 for unnormalized baselines. They then attach the tokenizer to Qwen3-0.6B-Base through a Language Extension Pipeline (LEP). LEP adds new vocabulary, initializes it from existing embeddings, freezes the original rows and retrains only the top layers. Evaluation loss falls from 8.28 to 2.43 in 800 steps. This makes adapting a small LLM to a new language cheap.

Key points

Fertility improves by about 18% with normalization (1.199 vs 1.35 tokens/word). The tokenizer is scored on fertility, characters per token and OOV rate.
Vocabulary targets are 80K for base runs and 150K for normalized variants, near Qwen3's 151,646. Frequency pruning at 95% or 99% cumulative coverage leaves roughly 42K or 76K tokens.
LEP adds only those Arabic SentencePiece tokens missing from Qwen3, after filtering out pieces with Latin, digits, Cyrillic or punctuation.
In LEP, new embeddings start as the mean of their Qwen3 subtoken embeddings. Gradient hooks zero out updates to the original embedding rows. Only layers 24–27 of the 28 are trainable, and the LM head stays frozen.
Training uses 100K ArabicText-Large samples with 2K for validation. The best ablation keeps Alif variants (Alif4) with LR=2e-4, giving a 71% loss drop. The authors suggest Hamza position carries case and word-origin cues.

Interesting ideas

I1. Normalization should be a switchable set of rules, tested by ablation, because the most aggressive merge was not best: keeping Alif forms helped the model since they carry grammatical signal. — For Mova (tokens): Treat each normalization (quotes, dashes, case, apostrophes in contractions) as an explicit, logged rule and keep any that hurts tagging or lemma accuracy switched off.
I2. A new vocabulary item can be seeded as the average of the old pieces it splits into, so it starts in a sensible spot instead of at random. — For Mova (tree): When gparse gets new word-piece or label embeddings, seed them from the mean of existing pieces to speed convergence and avoid random-start noise.
I3. Old embedding rows can be protected by zeroing their gradients while new rows and a few top layers train, which limits forgetting with a tiny budget. — For Mova (learn): Retraining on new silver data or a new UD dialect could freeze rows for established vocabulary and update only new items plus top layers, keeping regressions checkable.
I4. Pieces added to a vocabulary can be screened by script and character patterns so that only target-language material enters. — For Mova (tokens): Screen auto-induced lexicon or suffix entries with simple pattern filters (no digits, no foreign script, no stray punctuation) before they reach the tagger's lexicon.
I5. Vocabulary size can be set by pruning to a cumulative-frequency coverage level, which makes the size/coverage trade-off a single measurable knob. — For Mova (tokens): Size the morphology lexicon by coverage threshold and report how OOV rate changes at each cutoff.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages · Tokenization & segmentation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2512.18399 · arXiv 2512.18399

@misc{kashirskiy2025aratoken,
  title = {{AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3}},
  author = {Kashirskiy, Mark and Lipinski, Artiom and Makarov, Ilya},
  year = {2025},
  doi = {10.48550/arxiv.2512.18399},
  eprint = {2512.18399},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2512.18399}
}

References

6 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

nications of the ACM, 64(4):72–81, 2021. find
[3] Konstantin Dobler, Arij Riabi, Julia Kreutzer, and Benoît Sagot. Focus: Effective embedding initialization for monolingual specialization of multilingual models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13440– 13454, 2023. find
[4] Nizar Y Habash. Introduction to Arabic natural language processing. Springer, 2010. find
[5] Moussa Kamal Eddine, Antoine Tixier, and Michalis Vazirgiannis. Arabart: a pretrained arabic sequence-to-sequence model for abstractive summarization. In Proceedings of [13] Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 the Sixth Arabic Natural Language Processing IEEE international conference on acoustics, Workshop, pages 31–42, 2021. speech and signal processing (ICASSP), pages [6] Taku Kudo. Subword regularization: Improv-5149–5152. find
IEEE, 2012. ing neural network translation models with multiple subword candidates. arXiv preprint [14] Lane Schwartz, T Florian Jaeger, Ferenc Bunta, Liu Chen, William Martin, Allyssa De-arXiv:1804.10959, 2018. Giovine, and Kathryn Kohnert. On the right [7] Taku Kudo and John Richardson. Sentence-way to use unicode text normalization. arXiv piece: A simple and language independent preprint arXiv:1903.02974, 2019. subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 [15] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare Conference on Empirical Methods in Natu-words with subword units. In Proceedings of ral Language Processing: System Demonstra-the 54th Annual Meeting of the Association for tions, pages 66–71, 2018. Computational Linguistics, pages 1715–1725, [8] Benjamin Minixhofer, Fabian Paischer, and 2016. Navid Rekabsaz. Wechsel: Effective initialization of subword embeddings for cross-lingual [16] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, transfer of monolingual language models. In Timothée Lacroix, Baptiste Rozière, Na-Proceedings of the 2022 Conference of the man Goyal, Eric Hambro, Faisal Azhar, North American Chapter of the Association for et al. Llama: Open and efficient foun-Computational Linguistics, pages 3992–4006, dation language models. arXiv preprint 2022. arXiv:2302.13971, 2023. [9] El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. [17] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng-Arat5: Text-to-text transformers for ara-peng Li, Chengyuan Li, Dayiheng Liu, Fei bic language generation. arXiv preprint Huang, et al. Qwen2 technical report. arXiv arXiv:2109.12068, 2022. preprint arXiv:2407.10671, 2024. ↗
[10] Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, [18] Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ife-Go Inoue, Fadhl Eryani, Alexander Erdmann, oluwa Adelani, Khalid Almubarak, M Saiful and Nizar Habash. Camel tools: An open Bari, Lintang Sutawika, Jungo Kasai, Ahmed source python toolkit for arabic natural lan-Baruwa, et al. Bloom+1: Adding language guage processing. In Proceedings of the support to bloom for zero-shot prompting. In Twelfth Language Resources and Evaluation Proceedings of the 61st Annual Meeting of Conference, pages 7022–7032, 2020. the Association for Computational Linguistics, [11] Aleksandar Petrov, Emanuele La Malfa, Philip pages 11682–11703, 2023. Torr, and Adel Biber. Language model tokenizers introduce unfairness between languages. [19] Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. Advances in Neural Information Processing Llama beyond english: An empirical study Systems, 36, 2023. on language capability transfer. arXiv preprint [12] Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebas-arXiv:2401.01055, 2024. tian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 3118–3135, 2021. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kashirskiy-2025-aratoken-optimizing-arabic-tokenization-normalization/