📄 A rule-based approach to unknown word recognition in Arabic

Summary

The paper tackles out-of-vocabulary (OOV) verbs in Arabic. Instead of treating them only statistically, it treats rich derivational morphology as a source of meaning. Arabic verbs come from triliteral roots arranged into derived "Forms". Each Form carries a loose but recognisable semantic shift relative to the basic Form I, such as causative, reciprocal, reflexive/passive or "ask for X". Cahill encodes this in a PolyLex/PolyOrth lexicon written in DATR, which uses default inheritance. Stems are modelled as syllable structures: root consonants sit in onset and coda slots, and vowels depend on morphosyntactic features. From a seed lexicon of 100 verbs, the system generates the other Forms with a guessed meaning. It then matches these against unknown verb stems in the NEMLAR Written Arabic Corpus. Of 545 unknown verbs, 84 could be analysed as derivatives of known roots. In a manual check, roughly half the generated meanings were clearly related to the real ones. The work matters for applications that need semantic interpretation of rare words. It also suggests a route for regional varieties, whose lexicons could be built as default-overriding extensions of MSA.

Key points

• The framework is PolyLex (multilingual default-inheritance lexicon in DATR) plus PolyOrth (orthography via two-level FSTs for phoneme–grapheme mapping and spelling rules). It was earlier built for English, German and Dutch and is extended here to Arabic script, transliteration and phonology.
• MSA commonly uses ten verb Forms, while Classical Arabic has fifteen. Form meanings, adapted from Scheindlin (2007), are hints rather than rules. Examples include know → teach (II) → learn (V) and break → be broken (VII).
• The test used verb stems without prefixes or suffixes from the NEMLAR corpus. After merging inflectional variants there were 577 verb forms: 32 were already in the 100-verb lexicon, and 84 of the remaining 545 got a derivational analysis.
• Of those 84 analyses:
  • 23 had a dictionary meaning identical to Form I (e.g. ktatab ≈ katab "wrote").
  • 22 shifted meaning in the predicted way.
  • The rest, the largest group, deviated from the prediction, though usually not drastically.
• Motivation from Khoja (2001): Egyptian and Qatari news showed about twice the OOV rate of Saudi news, which is closer to MSA.

Interesting ideas

• I1. Approximate, overridable derivational semantics are useful: a rough default meaning for a derived form is still worth having when more specific lexical facts can replace it.
• I2. Morphology can be framed as generating candidate lexical entries rather than analysing at runtime: the lexicon proposes derived words with meanings, and a corpus confirms which exist.
• I3. Errors can be sorted into three bins (same meaning as base, predicted shift, unpredicted shift) instead of a single accuracy number.
• I4. Dialects can be modelled as default-inheriting deltas over a standard lexicon, so only the differences need specifying.
• I5. Emitting several candidate analyses and letting the downstream application choose is treated as the intended behaviour, not as failure.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): English OOV words with productive affixes (-er, re-, un-, -able) could get a default lemma plus a semantic relation to the base, marked as a default that a lexicon entry can override.
• I2 (learn): run affix rules over the known lexicon to propose new entries, then keep only those attested in corpora as silver lexicon extensions.
• I3 (learn): evaluate any derived-meaning or repair rule this way, so we can see whether a rule is wrong or merely unnecessary.
• I4 (world): compiled knowledge for domains or registers (e.g. fable vocabulary, British vs American usage) could inherit from a core lexicon and store only overrides, keeping provenance explainable.
• I5 (frames): for an unknown derived verb, pass several VerbNet-class hypotheses (e.g. causative vs reflexive reading) to frame building, and let the argument structure in the UD tree pick one.