📄 Arabic Tokenization System

Summary

Arabic writing often fuses several syntactic units into one orthographic word. Conjunctions, prepositions, the definite article and pronouns attach to stems with no visible separator, and they can stack and change shape through assimilation. A parser that builds trees from syntactic units therefore needs a tokenizer that knows Arabic morphology.

Attia describes a rule-based tokenizer built with Finite State Technology (lexc, Flag Diacritics). It serves an Arabic Lexical Functional Grammar on the XLE platform within the ParGram Project. He compares three designs:

• one transducer that both analyses and segments words;
• a separate guesser that treats any letter sequence as a possible stem, paired with a small clitic-only transducer merged into the main morphology;
• a segmenter limited to words the morphology already lists.

The guesser design is preferred because it is robust and modular. Its overgeneration is controlled later in the pipeline:

• the morphological transducer discards paths with invalid stems;
• a cashida mark separates split clitics from free words;
• a token filter labels rare splits and decomposed multiword expressions as "+undesired".

The outcome is a reusable pipeline architecture rather than a benchmark score. The available text reports no accuracy figures.

Key points

• A single Arabic word can carry up to four tokens. A verb may combine a conjunction, a complementizer, a stem and an object pronoun. In nouns, the definite article and the genitive pronoun cannot co-occur, and lexc enforces this with a shared Flag Diacritic (U.Def On/Off).
• "Main tokens" are separated by whitespace and punctuation, including the Arabic comma, question mark and semicolon, plus both Arabic and Hindi numerals. "Sub-tokens" are clitics and stems inside a word. Both kinds are delimited by '@'.
• The three models are combined analysis, guesser-based segmentation, and lexicon-dependent segmentation. Model 2 (the guesser) was chosen as optimal, even though a three-word sentence can already produce eight segmentations.
• Splitting creates new ambiguities. In kitabahum ("their book"), the separated hum could be a clitic, the free pronoun "they", or a noun meaning "worry". The proclitic ka ("as") and the enclitic ka ("you") also collide. The cashida-marker fix is credited to Ron M. Kaplan and Martin Forst.
• The token filter combines a custom list of legal but unlikely splits (e.g. ba'd → bi + 'add, baina → bi + yin) with a list imported from the MWE transducer (e.g. hazr al-tajawwul, "curfew"). Matching outputs are tagged rather than deleted.

Interesting ideas

• I1. Generate segmentations permissively with a guesser that accepts any stem, then let the lexicon/morphology prune paths with impossible stems, which keeps robustness on unknown words without accepting garbage.
• I2. After splitting, mark which side a clitic was attached on, so a detached piece cannot be confused with an identical free word.
• I3. Readings that are legal but rarely intended are flagged with a tag such as +undesired rather than removed, so control stays with later stages and nothing is silently lost.
• I4. The same multiword-expression list serves two purposes: it defines single tokens and it detects when a non-deterministic tokenizer has broken an expression into compositional pieces.
• I5. Clitic paths are removed from the main lexicon and moved into a small clitic-only transducer that is merged back in, so each module owns one job and the lexicon holds no duplicate entries.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): the en tokenizer could propose several splits for unseen or hyphenated forms and keep only those whose pieces get a valid lemma from morphology, with each discard logged as an explanation.
• I2 (tokens): English pieces like 's, 'd and n't could keep an attachment marker, so the tagger never treats a split 's like a standalone word and can still choose between possessive and "is/has".
• I3 (tree): the parser could receive soft penalties on such token paths, which keeps the explainable trace and lets a later step override the penalty when the context demands it.
• I4 (frames): prag could use one idiom/MWE inventory to block literal who-did-what-to-whom frames for fixed phrases like "kick the bucket" or "in spite of".
• I5 (tokens): the en crate could keep contractions and clitic-like pieces in a separate small lexicon combined with the main one, making each module simpler to test and debug.