📄 Handling Unknown Words in Arabic FST Morphology

Summary

Arabic morphological analysers depend on a lexicon, so newly coined or borrowed words are not analysed and cannot be lemmatized by lookup. Shaalan and Attia present what they describe as the first attempt to lemmatize Arabic unknown words. They ran MADA, which relies internally on SAMA, over a corpus of roughly 1.09 billion words from Arabic Gigaword (4th ed.) and Al-Jazeera news. Out-of-lexicon types were filtered with a spell checker and a frequency threshold. Each surviving word then passed through a pipeline. An XFST finite-state guesser lists every plausible segmentation into clitics, affixes and stem. The candidate whose morphological features best agree with MADA's top-scoring analysis is kept. A weighting formula then ranks the resulting lemmas for lexicographers to review. On a hand-annotated gold set of 1,310 types, lemmatization accuracy rises from a 45% baseline to 54%, or 63% when definite-article differences are ignored. This shows that a heavily overgenerating FST guesser can be useful in practice for growing a lexicon. The resulting unknown-word lexicon is released as open source.

Key points

• Filtering funnel:
  • 2,116,180 types had no SAMA analysis.
  • 208,188 of these were accepted by the MS Office 2010 Arabic spell checker.
  • 40,277 remained after keeping only types with frequency ≥10.
  • 1,310 were sampled for gold annotation (lemma, POS, include-in-dictionary judgement).
• The guesser treats a noun stem as any 2–24 non-numeric Arabic characters and a verb stem as 2–6 characters. It averages 12.6 analyses per word, compared with 2.1 for AraComLex. The gold lemma appears among its candidates 97% of the time.
• POS accuracy:
  • 45% when every word is tagged as a proper noun (baseline);
  • 60% with MADA alone;
  • 69% with "Voted POS Tagging" across forms that share a lemma.
• Lemmatization reduced 40,277 surface types to 18,399 lemmas, a 54% reduction.
• The ranking score combines three factors: sister-form count × 800, cumulative frequency of the sister forms (the two halved), and a POS factor that penalizes proper nouns.
  • Combined criteria: 78 good words in the top 100 and 15 in the bottom 100.
  • Frequency alone: 63 in the top 100 and 50 in the bottom 100.

Interesting ideas

• I1. Let a cheap rule-based guesser overgenerate every possible analysis, then pick the one whose features best match a statistical tagger's contextual prediction; candidate generation stays transparent while context decides.
• I2. Pool evidence across all surface forms that map to one lemma: if most of a lemma's forms carry a certain POS, override the minority tag.
• I3. Report an oracle ceiling (is the gold answer anywhere in the candidate set?) next to first-choice accuracy, so generation failures are separated from selection failures.
• I4. Put a spell-checker gate and a minimum-frequency threshold in front of lexicon growth, so typos are not mistaken for new vocabulary.
• I5. Rank new entries for human review by how many distinct inflected forms a lemma has, not only by raw frequency, and down-weight proper nouns whose frequency is inflated.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): An affix-stripping guesser for OOV English words in crate en could propose lemma/feature candidates, and the tagger's chosen features would select among them, with the rejected options kept as an explanation trail.
• I2 (tokens): Corpus-level voting over lemma families could correct sporadic tagger errors on rare or new words before they propagate into UD trees.
• I3 (learn): Evaluations of parsers or frame selection could log both candidate recall and top-1 accuracy, showing whether the fix belongs in the generator or the ranker.
• I4 (learn): Self-induced lexicon or repair-rule additions could require that a candidate survives a spelling filter and recurs often enough before it is accepted.
• I5 (world): When extending compiled knowledge, such as verb classes or ontology entries, from corpus discoveries, morphological productivity could be used as a signal of a genuine lexical item to prioritise curation.