📄 Generating an Arabic Full-form Lexicon for Bidirectional Morphology Lookup

Summary

Arabic morphology is hard to process. Stems change internally, broken plurals are irregular, and ordinary text usually omits short vowels. The authors build a resource that serves both analysis and generation of Arabic word forms. They start from an existing Arabic generator written in Morphe, which follows a stem-centred, lexeme-based framework. A Lisp routine expands lexical entries into their complete paradigms, producing a full-form lexicon. Each entry carries static properties (part of speech, idiosyncrasies) once, and each surface form adds only its dynamic features. The resulting pairs of surface form and analysis are then compiled into a minimal two-level finite-state transducer with a generic, scalable algorithm. A second, vowel-free transducer is built from the same data, so unvowelled input returns every vowelled reading. Lookup runs through DFKI's Java and C engines, which offer error-tolerant search and bindings for several languages. The paper reports feasibility, not accuracy: the system covers 65,000 analyses and processes about 4,000 words per second. The main value is a clean route from a linguist-maintained description to a compact, embeddable bidirectional analyzer.

Key points

• Lexicon source. The full-form lexicon comes from a Morphe-based Arabic generator (Leavitt, 1994). Features are split into static ones (shared per lexeme) and dynamic ones: voice, tense, mood, person, number and gender for verbs, and case, number and definiteness for nouns.
• Links missing elsewhere. The lexicon records broken plural → singular (e.g. rijaAlun → rajul), participles → their verb, and verbal nouns → their base. The authors note that the Arabic Tree Bank (ATB) and Xerox's analyzer lack such information; the ATB, for instance, has no number feature for broken plurals.
• Compilation. Each pair becomes a sequence of character-level and attribute-value labels, which are compiled into a minimal FST at a little over 4 bytes per transition. The same algorithm previously packed over 830,000 German full forms (6.5M+ analyses) into 1.2 MB.
• Lookup engines. DFKI provides a Java engine that is simple and thread-safe, and a faster C engine (SILO) with similarity search in the spirit of Oflazer (1996). Interfaces exist for Python, Perl, Prolog, LISP, Java via JNI and XML/RPC, and the engines are intended to feed HPSG parsing with LKB/PET.
• Evaluation data and speed. Tests used the 65,000-analysis lexicon and 34,590 analyzed forms extracted from the Arabic Treebank. Speed was about 4,000 words/s, and vowel-stripped queries such as Almdrs return several case readings.

Interesting ideas

• I1. Separate source from runtime. Keep morphology as readable rules for maintainers, expand them into an explicit full-form list, then compile that list into a minimal transducer for runtime. Each representation then serves one audience well.
• I2. Project one side to get a noisy-input analyzer. Deleting information from the surface side (here, vowels) yields an analyzer for underspecified input from the same data, at no extra annotation cost.
• I3. Store cross-lexeme derivational links. The lexicon connects forms to related lexemes (plural to singular, participle and verbal noun to verb) instead of only tagging surface affixes.
• I4. Use a weighted error metric for approximate lookup. Similarity search lets each application define how likely a given deviation is (typing, OCR, phonetic), rather than relying on plain edit distance.
• I5. Audit existing resources first. The authors deliberately listed which features other resources lack (e.g. ATB number marking) before fixing their own schema, so the lexicon would not need rebuilding later.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): crate en could expand its English inflection rules into a full table and compile it into one transducer for both lemmatization and inflection, where every analysis is an enumerable, inspectable path.
• I2 (tokens): building lowercase-, apostrophe-less or spelling-variant projections of the English lexicon would let the tagger recover full candidate analyses from sloppy student text.
• I3 (frames): linking nominalizations and participles to their verbs ("destruction" → destroy) would let prag assign VerbNet roles inside noun phrases, recovering who did what from nominal phrasing.
• I4 (tokens): a typo-weighted fallback lookup could map misspelled words in school tests to ranked lexicon candidates, and the chosen correction would stay visible as an explanation.
• I5 (learn): before generating silver data, compare the feature inventories of the witness treebanks and parsers to find attributes none of them supply, and add them to the schema up front.