📄 Capturing Out-of-Vocabulary Words in Arabic Text

Summary

The paper addresses how to detect foreign (transliterated, out-of-vocabulary) words inside Arabic text. Such words often appear in many spellings, and Arabic IR tools such as stemmers mishandle them. The authors contrast assimilated loanwords, which usually have one stable spelling, with freshly transliterated terms and names, which vary widely. They compare three families of detectors:

• Lexicon lookup, where a word missing from an Arabic dictionary is flagged as foreign.
• Morphological pattern matching, which strips affixes and checks whether the remaining stem fits a root pattern; this was built on a modified Khoja stemmer.
• Character n-gram profiles, learned from foreign words.

The test data were word lists that were hand-labelled as Arabic, foreign or misspelled. A set of linguistically motivated veto rules, which block the foreign label, raised precision for every method. The lexicon approach based on Microsoft Office proofing tools (OLA) performed best overall. Results on a second, unseen newspaper word list were weaker but showed the same trends. Reliable detection matters because spelling-variant expansion or normalisation should only be applied to genuinely foreign words.

Key points

• The name Milosevic appeared in 28 different Arabic spellings in the TREC 2002 Arabic collection.
• After the veto rules, OLA reached about 69% precision and 71% recall on the first data set. The rules lifted precision by roughly 10–25% across approaches, at a small cost in recall.
• The verification set held 23,466 unique words from Dar-al-Hayat: 22,800 Arabic, 536 foreign and 130 misspelled. After the rules, OLA reached P 0.474 and R 0.572, and BLA had the highest precision at 0.502.
• When given 3,046 known foreign words, the pattern approach labelled 2,017 correctly and wrongly called 1,029 Arabic. Missing diacritics let foreign words accidentally fit Arabic patterns.
• Lexicons are imperfect in both directions. They lack many inflected forms, yet about 200 foreign words were already in the OLA lexicon. The trigram method (TRG) did worst, partly because Arabic and foreign words share many trigrams.

Interesting ideas

• I1. Gold labels used three classes (native, foreign, misspelled), so errors could be traced to each confusion rather than lumped as "unknown."
• I2. Rather than tuning one classifier, the authors added cheap, explainable veto rules that override the foreign label, such as a native definite-article prefix or stacked native suffixes.
• I3. They caught missing-space joins by spotting a character that can only end a word sitting in the middle of a token, then checking that both halves are long enough.
• I4. Pattern matching reverses derivation: affixes are stripped and the leftover stem is tested against known templates, so a word is accepted as native only if it can be explained morphologically.
• I5. Rules were developed on one word list and then re-checked on a fresh corpus, which exposed the expected precision drop honestly.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Unknown-word handling in the en crate could split OOV tokens into likely-foreign, likely-typo and rare-native classes, each with its own lemma and tag fallback.
• I2 (tokens): Native English morphology cues (productive affixes such as -ness or un-) could act as explicit, logged vetoes before a token is marked as a foreign or name OOV.
• I3 (tokens): The tokenizer could flag run-together tokens when an internal split yields two known words of sufficient length, and record the split as a traceable repair.
• I4 (world): Compiled lexical knowledge could treat a word as "explained" only if it can be derived from known roots and affixes, leaving unexplained items as candidate names or borrowings for the entity layer.
• I5 (learn): Self-induced repair rules should always be re-scored on a held-out text source before adoption, and any loss in transfer should be reported.