Elamur Research⌕
☰
home / papers / 2026 / Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs

Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs

Summary

The paper asks whether LLMs and their tokenizers actually model Arabic root-and-pattern morphology, where consonantal roots interleave with templatic patterns, or whether they mostly rely on memorized surface forms. The authors compare tokenizers from seven models: GPT-4, GPT-4o, LLaMA 3, Qwen 3 and Cohere (multilingual), plus Fanar and Allam (Arabic-centric). They score each tokenizer against gold morphological segmentation in two corpora, ATB3 (Modern Standard Arabic) and the BOLT Egyptian Arabic treebank (dialectal). The CAMeL Tools MLE segmenter and Farasa serve as linguistically informed baselines. The authors also build a new test set that asks models to apply derivational patterns and affixes to real and invented (nonce) roots. The main result is that how well a tokenizer matches morpheme boundaries does not predict generation quality. GPT-4 has poorly aligned, high-fertility tokenization yet ranks second. Fanar, the only model with a morphology-informed tokenizer, generalizes weakly. This matters because it challenges the common assumption that morphological tokenization and low fertility drive downstream competence.

Key points

  • Gold data: ATB3 (LDC2010T08) and BOLT Egyptian Arabic (LDC2021T12).
    • Diacritics, punctuation, numbers and Latin script were stripped, so evaluation runs on undiacritized words.
    • After cleaning, ATB3 has about 292.5K words and BOLT about 128K.
  • Two alignment views:
    • boundary correspondence between tokens and morphemes;
    • morphological integrity, meaning whether whole morphemes stay inside a single token.
  • Generation test design:
    • real roots and nonce roots;
    • 13 derivational patterns plus an affix-building task;
    • zero-shot and one-shot prompts, in both English and Arabic;
    • temperature 0.6.
  • Strongest models: GPT-4o is best and stable at roughly 95–97% on real and nonce roots across all prompt settings. GPT-4 reaches about 95% on real roots with English one-shot prompts.
  • Weaker models: LLaMA 3 scores 0–10% on nonce roots. Fanar falls to about 1% on real roots with an Arabic zero-shot prompt, despite its MorphBPE-based tokenizer.

Interesting ideas

  • I1. Pairing real roots with invented nonce roots separates true productive rule use from recall of familiar forms, which is a clean memorization control for any morphology test. — For Mova (tokens): Build a wug-style set of nonce English verbs and nouns ("to blick", "blicked", "blickers") and check that the tagger, morphology and lemmatizer in crate en still produce correct lemmas and features.
  • I2. Alignment is scored on two separate axes: whether boundaries match, and whether morphemes stay whole within one token, because a single score blurs partial overlaps with real boundary errors. — For Mova (tokens): Report the morphological segmenter's errors split into boundary misplacement versus morpheme fragmentation, so each failure points to a specific repair.
  • I3. A tidy structural score (tokenizer–morpheme alignment) failed to predict behavioral performance, a reminder to validate any intrinsic metric against the downstream task before trusting it. — For Mova (learn): Before adopting an intrinsic metric for silver data or witness-parser agreement as a training signal, check that it actually correlates with tree and answer accuracy, using negative controls.
  • I4. Every model was run across a grid of prompt language and shot count, which exposed large swings for some models and stability for others. — For Mova (learn): When ranking LLM teacher tiers, run each teacher across several prompt variants and prefer teachers whose labels stay stable, not just those with a high best-case score.
  • I5. The authors state openly that removing diacritics makes some templatic contrasts invisible, recording what a normalization step hides from evaluation. — For Mova (tokens): Document which distinctions each text-normalization step in crate en erases (case, apostrophes, hyphenation), so evaluation does not credit or blame contrasts that the input no longer contains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

1 paper of our library cites it (of the 140884 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.15773 · arXiv 2603.15773

@misc{alakeel2026morphemes,
  title = {{Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs}},
  author = {Alakeel, Yara and Qwaider, Chatrine and Aldarmaki, Hanan and Alqahtani, Sawsan},
  year = {2026},
  doi = {10.48550/arxiv.2603.15773},
  eprint = {2603.15773},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.15773}
}

References

38 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ummar Abbas et al. 2025. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944. ↗ our card
  2. Josh Achiam et al. 2024. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗ our card
  3. Arash Ahmadian et al. 2025. Command a: An enterprise-ready large language model. arXiv preprint arXiv:2504.00698. ↗
  4. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, et al. 2024. Tokenizer choice for llm training: Negligible or crucial? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3907–3924. find
  5. Catherine Arnett and Benjamin Bergen. 2025. Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607–6623, Abu Dhabi, UAE. Association for Computational Linguistics. find
  6. Catherine Arnett, Marisa Hudspeth, and Brendan O’Connor. 2025. Evaluating morphological alignment of tokenizers in 70 languages. In Proceedings of the ICML 2025 Tokenization Workshop (TokShop). International Conference on Machine Learning. find
  7. Ehsaneddin Asgari, Yassine El Kheir, and Mohammad Ali Sadraei Javaheri. 2025. Morphbpe: A morpho-aware tokenizer bridging linguistic complexity for efficient llm training across morphologies. arXiv preprint arXiv:2502.00894. ↗
  8. M Saiful Bari, Yazeed Alnumay, Norah Alzahrani, Nouf Alotaibi, Hisham Alyahya, AlRashed Al-Rashed, Faisal Mirza, Shaykhah Alsubaie, Hassan Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman I Alsubaihi, Maryam Al Mansour, Saad Hassan, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, and Haidar Khan. 2025. Allam: Large language models for arabic and english. In International Conference on Learning Representations, volume 2025, pages 34179–34214. find our card
  9. Kenneth R Beesley and Lauri Karttunen. 2003. Finite-state morphology: Xerox tools and techniques. CSLI, Stanford, pages 359–375. find
  10. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617– 4624, Online. Association for Computational Linguistics. find
  11. Anh Dang, Limor Raviv, and Lukas Galke. 2024. Morphology matters: Probing the cross-linguistic morphological generalization abilities of large language models through a wug test. In 13th edition of the Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2024), pages 177–188. Association for Computational Linguistics (ACL). find
  12. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074. find
  13. Philip Gage. 1994. A new algorithm for data compression. C Users J., 12(2):23–38. find
  14. Bar Gazit, Shaltiel Shmidman, Avi Shmidman, and Yuval Pinter. 2025. Splintering nonconcatenative languages for better tokenization. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22405–22417, Vienna, Austria. Association for Computational Linguistics. find
  15. Aaron Grattafiori et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  16. Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Morgan & Claypool Publishers. find
  17. Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 385–393, Dublin, Ireland. Association for Computational Linguistics. find
  18. Valentin Hofmann, Leonie Weissweiler, David R Mortensen, Hinrich Schütze, and Janet B Pierrehumbert. 2025. Derivational morphology reveals analogical generalization in large language models. Proceedings of the National Academy of Sciences, 122(19):e2423232122. find
  19. Binyuan Hui et al. 2025. Qwen2.5 technical report. arXiv preprint arXiv:2409.12186. ↗ our card
  20. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. ↗ our card
  21. Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, and Lonneke Van Der Plas. 2025. Evaluating morphological compositional generalization in large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. find
  22. Haris Jabbar. 2024. Morphpiece : A linguistic tokenizer for large language models. arXiv preprint arXiv:2307.07262. ↗
  23. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. find
  24. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics. find
  25. Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp. find
  26. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association. find our card
  27. Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, et al. 2024. Byte latent transformer: Patches scale better than tokens. arXiv preprint arXiv:2412.09871. ↗
  28. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. Advances in neural information processing systems, 36:36963–36990. find
  29. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics. find our card
  30. Craig W Schmidt, Varshini Reddy, Chris Tanner, and Yuval Pinter. 2025. Boundless byte pair encoding: Breaking the pre-tokenization barrier. arXiv preprint arXiv:2504.00178. ↗
  31. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. find
  32. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics. ↗
  33. Jean Seo, Jaeyoon Kim, SungJoo Byun, and Hyopil Shin. 2025. How does a language-specific tokenizer affect llms? arXiv preprint arXiv:2502.12560. ↗
  34. Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schuetze, Kemal Oflazer, and David Mortensen. 2023. Counting the bugs in ChatGPT’s wugs: A multilingual investigation into the morphological capabilities of a large language model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6508–6524, Singapore. Association for Computational Linguistics. find our card
  35. Leonie Weissweiler, Kyle Mahowald, and Adele Goldberg. 2025. Linguistic generalizations are not rules: Impacts on evaluation of lms. arXiv preprint arXiv:2502.13195. ↗
  36. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. 7. find
  37. Language Resource References Maamouri, Mohamed and Bies, Ann and Buckwalter, Tim and Mekki, Wigdan. 2010. Arabic Treebank: Part 3 v 3.2 LDC2010T08. ISLRN 770-467-034-042-0. Web Download. Philadelphia: Linguistic Data Consortium. find
  38. Maamouri, Mohamed and et al. 2021. BOLT Egyptian Arabic Treebank - Conversational Telephone Speech LDC2021T12. ISLRN 430-645-589-448-0. Web Download. Philadelphia: Linguistic Data Consortium. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alakeel-2026-morphemes-borders-evaluating-root-pattern/