📄 Arabic Word Generation and Modelling for Spell Checking

Summary

The paper addresses the lack of an adequate open-source spell-checking lexicon for Arabic. Existing free resources such as Ayaspell (about 300K forms) and Arabic-Spell (about 900K forms) cover only a small part of the language's highly productive inflection and cliticization. The authors used the open-source finite-state morphological transducer AraComLex in generation mode to enumerate roughly 13 million surface forms. They filtered these, together with other word lists, through the Microsoft Spell Checker, which produced "AraComLex Extended", a list of about 9.3 million accepted words released under GPLv3.

The accepted and rejected forms then served as training data for two character-level tri-gram models, one for valid words and one for invalid ones. Comparing the two gives an error detector for words not in the list, reported at 98.2% precision at full recall.

For correction, the system combines finite-state edit-distance candidate search, Noisy Channel ranking, Arabic-specific letter-confusion costs and rules for splitting merged words. It clearly beats Hunspell but stays below MS Spell Checker. The work matters because it supplies a large free lexicon and a detector that generalizes beyond any fixed list.

Key points

• AraComLex (30,587 lemmas) generated 12,951,042 forms. MS Spell Checker accepted 8,783,858 of them (68%). The merged, deduplicated list has 9,306,138 accepted words, and 5,841,061 forms were rejected.
• On a 400,000-word news test set kept separate from Gigaword, AraComLex Extended scored 0.9840 coverage, 0.9998 accuracy and 0.9999 f-measure. Ayaspell's 292,464-word list scored 0.5529 coverage.
• Detection uses two character tri-gram models built with SRILM. The abstract reports 98.2% precision at 100% recall, while the introduction gives 97%.
• First-choice correction accuracy on 1,799 misspelled words:
  • MS Spell Checker: 80.54%
  • Hunspell with Ayaspell: 45.64%
  • edit distance plus Noisy Channel: 68.2%
  • with added heuristics: 71.3%
  • with post-processing of unknown words: 75%
• Of 227 words that received no candidates, 63% were two or more words run together. Regex rules targeting 7 frequent particles and words fixed 102 of the 105 affected cases.

Interesting ideas

• I1. Instead of collecting attested words from a corpus, run a bidirectional morphological analyser backwards to enumerate every form it licenses, then filter the output with an external judge; this finds forms no corpus contains.
• I2. Train one character n-gram model on accepted forms and another on rejected forms, then classify a new word by which model scores it higher; this captures legal letter clusters without needing an explicit rule for each constraint.
• I3. Lower the edit cost for substitutions within known confusion groups of letters, so that the ranking reflects real error patterns rather than treating every letter change as equal.
• I4. Scale the allowed number of edits with word length, setting the threshold from the corpus distribution of word lengths, so short words are not over-corrected and long words are not under-corrected.
• I5. Examine the cases where no candidates are produced; many turned out to be merged words that a few split rules repair, which shows that diagnosing failure classes can pay off more than tuning the ranker.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Generating all inflections from the crate en lemma lexicon would give a closed list of valid forms to check the tagger and lemmatizer against.
• I2 (tokens): A valid-versus-invalid character model could flag unknown tokens as likely typos or as plausible new words before morphology tries to analyse them.
• I3 (learn): Self-induced repair rules could weight candidate fixes by observed confusion classes, such as frequent tag or attachment swaps, rather than by flat counts.
• I4 (tokens): Tokenizer normalisation could use length-dependent edit budgets when mapping noisy tokens to lexicon entries.
• I5 (learn): Evaluation could routinely group zero-output parser and reader failures into classes, so that cheap targeted rules are written before any retraining.