📄 Using Stemming in Morphological Analysis to Improve Arabic Information Retrieval

Summary

Arabic words often fuse a core form with attached prefixes (articles, prepositions, conjunctions) and suffixes (mostly pronouns). As a result, a query word rarely matches the surface form found in a document. The authors present the Arabic analyzer of the LIC2M cross-lingual search engine, built in the European ALMA project, and focus on its clitic stemmer.

The stemmer runs on tokens the lexicon does not recognize. It normalizes vowel marks and letter variants, then lists every possible proclitic/enclitic split. Each remaining radical is checked against a large full-form lexicon, with re-write rules tried when the direct lookup fails. Finally, segmentations whose component tags are incompatible are discarded. The surviving candidates go into a word graph, and a trigram/bigram POS tagger later chooses among them.

Indexing only the simple forms, and treating clitics as empty words, raised precision by about 41% for monolingual retrieval and about 27% for cross-lingual retrieval. This matters for any language where agglutination splits one concept across many surface strings.

Key points

• The Arabic full-form lexicon was produced by an automatic inflector. It holds 3,164,000 entries derived from 114,000 lemmas and is stored as a finite-state automaton. Each entry is vowelled and linked to its unvowelled versions.
• There are two clitic dictionaries: 77 proclitic entries and 65 enclitic entries. They include valid proclitic combinations, such as conjunction + preposition + article.
• The idiom recognizer uses 482 contiguous expression rules written over lemmas. The POS tagger's n-gram matrices come from hand-tagged corpora of 13,200 Arabic words, 239,000 English words and 25,000 French words.
• In the evaluation, 100 Arabic questions were run against 50 non-parallel documents per language (Arabic, English, French), with relevance judged manually. The documents cover sustainable development, water and eco-tourism, and two runs were compared: stemmer on and stemmer off.
• Monolingual results beat cross-lingual ones, which the authors attribute to bilingual dictionary lookup depending on both lemma and POS. The findings agree with the light-stemming results of Larkey et al. (2002).

Interesting ideas

• I1. The stemmer does not commit to one split. It keeps every lexicon-verified segmentation as a parallel path in the word graph, and the contextual tagger later decides which path survives.
• I2. When a stripped radical is missing from the lexicon, a small set of re-write rules repairs the stem and the lookup is retried. Each split is accepted only if the tags of prefix, stem and suffix are compatible.
• I3. Multiword expressions are triggered by a lemma plus a tag rather than a surface string. This makes the rules robust to spelling and inflection variants, and the matched expression becomes a single typed node, such as a month name.
• I4. The tagger tries attested trigrams first and falls back to bigrams only at gaps. If neither applies, the word is deliberately left ambiguous instead of being forced to a guess.
• I5. Each document gets two questions: one taken verbatim from its text and one only topically related. A single module is then toggled on and off over the same set.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): English ambiguities such as "'s" (possessive vs. "is"/"has") or hyphenated forms could stay as alternative lattice paths until tagging. The choice would then be recorded and explainable, not hard-coded in the tokenizer.
• I2 (tokens): lemmatizing unknown English inflections could follow the same "strip, repair spelling, re-check lexicon, check tag agreement" loop, giving an auditable trace for each lemma.
• I3 (tokens): MWE and named-date recognition could fire on lemma+UPOS triggers with context checks. Inflected or oddly spelled variants would then still collapse into one node before parsing.
• I4 (learn): silver-data pipelines could likewise mark tokens as unresolved when no witness evidence covers them. This keeps noise out of training and flags cases for an LLM teacher tier.
• I5 (answer): QA on fables could pair a literal question with a paraphrased one per story and run module ablations. This would separate surface matching from genuine understanding.