📄 Systematic Verb Stem Generation for Arabic

Summary

Arabic verbs are hard to analyse because root letters are often replaced, merged, reordered or dropped when a word is built, so guessing the root from the surface letters is unreliable. The authors reverse the usual approach. Instead of analysing words, they generate every valid verb stem from known roots, the way native speakers build forms. Their database links all verb roots from the Al-Waseet dictionary to morphosemantic and morphosyntactic templates, plus tables of subject affixes. Stems are produced by slotting radicals into templates and then running ordered Perl regular-expression rewrites, which act as finite-state transducers for morphophonemic changes. Placeholder markers record which letters come from the root and which come from the template. This keeps every transformation traceable and reversible. A compact coding scheme for the rules cuts generation time sharply. The resulting stems, each stored with its full analysis, are loaded into a searchable index. Analysis then becomes a lookup: strip non-altering clitics and match the remainder against pre-analysed stems. This gives a certain root and pattern instead of a closest-match guess.

Key points

• The database has three tables, RootsList, Template and MainDictionary, plus a subject-affix table. Roots have up to four radicals, written F, M, L and Q, and the paper uses Buckwalter transliteration.
• Transformations are sequential regex substitutions. Each intermediate template can be decomposed back into the current radicals and template, and the placeholders are deleted at the end.
• The optimised surface-template plus affix-code scheme generated about 2.2 million stems in 720 seconds. The original regex cascade needed about 18,000 seconds for the same output.
• A root yields about 2.5 dictionary entries on average. Each entry yields 117 affixed stems, so a root gives about 300 stems on average. Productive roots such as {k,t,b} yield 13 entries, or about 1,500 stems.
• The output is stored in a B-Tree database file and exposed through a web CGI that lists all affixed stems of a given root. Object pronouns and similar clitics are agglutinative and are handled by simple stripping.

Interesting ideas

• I1. The authors precompute every inflected form together with its analysis, so analysis becomes an exact lookup rather than a heuristic decomposition. This removes uncertainty whenever the form space is finite.
• I2. Marker symbols are inserted before the letters that came from the root, so every rewrite step knows the origin of each character, and any intermediate state can be split back into root and pattern.
• I3. The authors model morphophonology as a short ordered chain of small rewrite rules, each producing an inspectable intermediate form, rather than one monolithic rule.
• I4. After the rule cascade was validated, it was compiled into a compact surface template plus positional affix codes. This trades flexibility for roughly 25 times faster bulk generation.
• I5. Clitics that cause no spelling change are peeled off first, so that only the true stem has to be matched. This splits a hard problem into a trivial step and a lookup step.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Generate a full inflection table from the lexicon in crate en, so the lemmatizer can return a lemma with certain provenance and fall back to rules only for unknown words.
• I2 (back): During generation, tag which characters come from the lemma and which from the suffix (for example, doubling in "stopped" or "y→i" changes), so each surface spelling can be explained and inverted.
• I3 (learn): Self-induced repair rules could be applied as an ordered cascade that logs each intermediate tree, making it easy to see which rule caused a regression.
• I4 (tokens): Keep readable morphology rules for development, but compile them into flat template tables for fast tokenizing of large silver corpora.
• I5 (tokens): Separate non-altering clitics such as 's, n't and 'll before lemma lookup, and record the split so the UD tree and frames can cite it.