📄 Language Model Based Arabic Word Segmentation

Summary

The paper tackles splitting Arabic words into their meaningful pieces. Arabic packs prepositions, articles, pronouns and agreement markers into a single orthographic word. The authors model every word as zero or more prefixes, one stem, and zero or more suffixes. A trigram language model over morphemes then picks the most likely split of a whole sentence. The model starts from a small hand-segmented LDC corpus of about 110K words and a table of known affixes. It is then grown without supervision: new stems are harvested from a 155-million-word raw corpus, judged by how closely they resemble stems already seen in training, and the model is retrained on the larger vocabulary. On a held-out set of 28,449 tokens, the segmenter reaches roughly 97% exact-match accuracy. Adding part-of-speech information in a joint model lowers the error rate a little further. The method matters because it needs only a modest seed corpus, an affix list and raw text, so it could transfer to other heavily inflected or agglutinative languages.

Key points

• Resources: an LDC hand-segmented corpus of about 140K tokens, split into training and development test data, plus a 155M-word unsegmented Arabic corpus. The affix table has 51 prefixes and 72 suffixes taken from LDC data, with 14 more prefixes and 122 more suffixes from IBM-Egypt.
• Error rate falls as components are added. With 110K training words, the plain trigram LM has 11.0% error, adding the prefix-suffix filter brings it to 5.5%, and adding new stems brings it to 2.9%. With only 10K words, the same steps go from 26.0% to 8.5%.
• Stem acquisition sharply cuts unknown words. In the 110K setup, LM vocabulary grows from 8,196 to 25,306 entries, and the out-of-vocabulary ratio drops from 5.8% to 1.9%.
• Unseen stems are handled through an UNKNOWN class. Its probability is multiplied by a tiny constant, UNK_Fraction = 1e-9, chosen empirically.
• A joint segmentation and POS model reduces the best error rate from 2.9% to 2.6%, roughly a 10% relative gain.
  • It targets cases that a morpheme LM alone cannot resolve, such as Alywm ("today" or the name Al-Youm vs. "the day").
  • It also targets over-split foreign names like bwtyn (Putin).

Interesting ideas

• I1. Build candidate splits only from a fixed affix table, but also break each matched compound affix into its parts (wA# → w# A#), then let a sequence model choose among the candidates; this keeps the search small while still reaching correct analyses that a flat lookup would miss.
• I2. Give out-of-vocabulary stems a shared UNKNOWN class with a heavily discounted probability, so new words can still be segmented without overwhelming known analyses.
• I3. Start from a small gold seed, use it to grow the lexicon from a huge raw corpus, then retrain on the expanded data; most of the gain shows up when the seed is tiny.
• I4. Sort errors into a few interpretable buckets (unknown stem, one notorious ambiguous token, everything else) and track how the shares shift as training size grows; this reveals which component should be improved next.
• I5. When a split depends on word class, score segmentation and POS tags together rather than one after the other.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): English clitics and contractions ("won't", "y'all'd", "'s") could be split this way, with candidates scored in context instead of resolved by fixed rules.
• I2 (tokens): a calibrated unknown-lemma class could let the morphology step fall back gracefully on rare words while keeping its decision traceable.
• I3 (learn): this matches the silver-data loop, and measuring the gain at several seed sizes (10K to 110K) would show where silver data pays off most.
• I4 (learn): evaluation reports could include such bucketed error tables, so repair rules and teacher effort go where the remaining errors actually sit.
• I5 (tokens): English "'s" (is / has / possessive) could be decided jointly with the tagger, avoiding errors that propagate into the UD tree.