📄 Effective Stemming for Arabic Information Retrieval

Summary

The paper looks at how to reduce Arabic words to index terms for monolingual information retrieval. Arabic words often carry several attached pieces: prepositions, person markers, inflectional endings and pronoun clitics. A common light stemmer cuts only a few frequent affixes from each end, so related word forms can still end up as different index terms. The authors propose a linguistic-based stemmer that models a word as {antefix, prefix, core, suffix, postfix} and tries to recover the core. Rules list every possible decomposition of a word. When several are possible, the stemmer picks the candidate stem that occurs most often across the 523,359 distinct words of the Arabic TREC collection. Both stemmers were tested on 383,872 AFP newswire documents with the TREC 2001 and 2002 topics. Retrieval used a KL-divergence unigram language model with Jelinek-Mercer smoothing. The core-based stemmer gave higher MAP on both topic sets, 0.3107 against 0.2868 on the merged set. This suggests that grouping word forms by their semantic core helps Arabic IR more than shallow truncation does.

Key points

• Data: Arabic TREC collection of 383,872 AFP articles (May 1994 to December 2000) with 523,359 distinct words. Topics were TREC 2001 (25) and TREC 2002 (50), using the title and description fields.
• MAP, linguistic-based vs light stemming:
  • TREC 2001: 0.3326 vs 0.3220
  • TREC 2002: 0.2828 vs 0.2671
  • merged 75 topics: 0.3107 vs 0.2868
• Recall results were mixed:
  • TREC 2001: 2704 vs 2664 of 4122 relevant documents found
  • TREC 2002: 4301 vs 4443 of 5909, so light stemming retrieved more here
  • merged: 7181 vs 7121 of 10,031
• Preprocessing was applied to both documents and topics:
  • alif variants unified, final ى mapped to ي, final ة mapped to ه, and ءي mapped to ئ
  • tatweel, shadda and diacritics removed
  • a 413-word stoplist applied
• Error analysis showed two light-stemmer failures. It misses one-letter suffixes such as ي, and it wrongly strips ان in words where those letters belong to the core. Corpus-frequency selection fixes the second case.

Interesting ideas

• I1. Generate all rule-licensed decompositions of a word, then keep the one whose stem is most frequent as a word in the corpus. Because a true stem should itself appear in text, this resolves ambiguous affixes without hand-written disambiguation rules.
• I2. Treat a word as an ordered sequence of slots (outer clitic, inner affix, core, inner ending, outer clitic) rather than stripping one piece from each end. This catches stacked affixes that a shallow stripper leaves behind.
• I3. Name both failure modes of conflation. Too little grouping splits related words and hurts recall, while too much grouping merges unrelated words and hurts precision. Then check examples of each kind.
• I4. Apply exactly the same orthographic normalization to queries and documents before matching, so spelling variants that do not change meaning stop blocking matches.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The en lemmatizer could rank candidate lemmas for unknown or derived forms by how often each appears as a standalone token. The chosen split and its frequency would be logged as the explanation.
• I2 (tokens): English derivational morphology (un-…-able-ity, re-…-ed) could use a similar slot template. Each segment would carry an explicit role that later stages can inspect.
• I3 (learn): When evaluating lemma or concept merging, report separate counts of wrong splits and wrong merges, with example lists. This would reveal hidden trade-offs that a single accuracy number masks.
• I4 (answer): Questions, texts and lexicon entries could all pass through one shared normalizer for apostrophes, quote marks, hyphenation and British/American spelling before readers match entities.