🌿 Stemming
Stemming is the reduction of inflected or derived word forms to a common stem, usually by stripping affixes with rules or statistics, so that variants such as "connect", "connected" and "connection" are treated as one term in search and text analysis.
What it is
Words appear in many shapes. A plural, a past tense or a derived noun each counts as a separate word form. A program that matches text character by character treats "running" and "runs" as unrelated strings. Stemming fixes this by mapping related forms onto one shared key, the stem.
A stem is whatever is left once the affixes a stemmer recognizes have been removed. It does not have to be a real word. "Relational" and "relate" can both become "relat". What matters is that related words reach the same stem and unrelated words do not. A lemma, by contrast, is always a real word. It is a dictionary headword chosen through morphology and often through part of speech (POS).
Stemming is one of the oldest preprocessing steps in information retrieval. It sits between tokenization and the construction of an inverted index. Neighbouring tasks rely on it too. Translation metrics such as METEOR need language-specific stemmers to credit matches between morphological variants. That dependency is one reason such metrics were long unavailable for most target languages (Denkowski & Lavie 2014).
The key idea
Rules over the end of the word
A classic English stemmer reads the word from the right and applies ordered rewrite rules of the form "if the word ends in this suffix and the remainder meets this condition, replace the suffix with that one." Porter (1980) built his conditions on a measure of how much word remains once the suffix is removed:
[C] (VC)^m [V]
In this pattern, C stands for a run of consonants and V for a run of vowels. Square brackets mark optional parts. The exponent m, the "measure", counts how many vowel–consonant sequences the remaining stem contains. Many rules apply only when m is above some threshold, which prevents short words from being reduced to nothing.
Porter's algorithm on a few words
The algorithm runs its steps in a fixed order:
- Step 1a deals with plurals. "caresses" becomes "caress" and "ponies" becomes "poni".
- Step 1b removes -ed and -ing, then tidies the result. "hopping" loses -ing and its doubled consonant, giving "hop".
- Steps 2 and 3 map long derivational suffixes onto shorter ones. For "generalization", Step 2 gives "generalize" (-ization to -ize), and Step 3 gives "general" (-alize to -al).
- Step 4 removes residual suffixes such as -al, -ance or -ate when the remaining stem has a measure above one. "general" becomes "gener".
- Step 5 removes a final -e and reduces a double -l under measure conditions. For "relational", Step 2 has already rewritten -ational as -ate, giving "relate". Step 5 then drops the -e, leaving "relat".
Two kinds of error
A stemmer can fail in two opposite directions. Over-stemming joins words with different meanings. Porter's rules send "university" and "universe" to the same stem. Under-stemming leaves related words apart, as with "alumnus" and "alumni". Arabic tweets show both kinds. In under-stemming an affix stays attached. In over-stemming a stem letter is taken for an affix, such as an initial "w" read as the conjunction (Albogamy & Ramsay 2016).
Stemming versus lemmatization
| Stemming | Lemmatization | |
|---|---|---|
| Output | stem, may not be a word ("relat") | dictionary form ("relate") |
| Knowledge used | affix rules or statistics | lexicon, morphology, often POS |
| Irregular forms ("went") | usually missed | mapped to "go" |
| Speed and cost | fast, cheap | slower, needs resources |
| Typical errors | over- and under-stemming | lexicon gaps, POS mistakes |
History and founding papers
Lovins (1968) published the first widely used English stemmer, which matched words against a long list of endings. Porter (1980) replaced that list with a short cascade of conditional rules, and his algorithm became the standard baseline. Paice (1990) developed the Lancaster (Paice/Husk) stemmer, which applies an iterative rule table and is known for aggressive conflation. Porter (2001) later introduced Snowball, a small language for writing stemmers. It came with stemmers for many European languages and a revised English version often called "Porter2".
Arabic changed the problem. Its words are built by applying patterns to consonantal roots and adding prefixes or suffixes (El-Defrawy et al. 2015). The other Semitic languages have a similar root-and-pattern structure. A single Arabic orthographic word can also combine a core with attached articles, prepositions and conjunctions in front and pronouns at the end. A query term therefore seldom matches a document's surface form exactly (Semmar et al. 2006). Two approaches emerged:
- Light stemming, associated with Larkey and colleagues, removes a small set of frequent prefixes and suffixes (Kadri & Nie 2006; Semmar et al. 2006).
- Root stemming, of which the Khoja stemmer is the best-known rule-based example, tries to recover the consonantal root (El-Defrawy et al. 2015).
A survey of Arabic language technology places statistical and stemming methods after the morphological analyzers of the 1980s and before embeddings and neural encoders (Al-Khalifa; & Alam 2025).
How it is used today
Search: recall and precision in Arabic retrieval
In an index, stemming usually increases recall, because more documents match a query. It can lower precision when over-stemming joins unrelated terms. For Arabic, whose words carry attached clitics, the balance can favour precision as well. In one cross-lingual engine, the index was built on simple forms only, with clitics handled as empty words. Precision was about 41% higher for monolingual retrieval and about 27% higher for cross-lingual retrieval than in the same engine with its stemmer turned off (Semmar et al. 2006).
The Text REtrieval Conference (TREC) Arabic newswire collection allows a direct comparison of designs. One stemmer models words as antefix, prefix, core, suffix and postfix, and resolves ambiguity by corpus frequency. It achieved a higher mean average precision (MAP) than light stemming on the merged topic set (0.3107 against 0.2868). Recall varied between topic years (Kadri & Nie 2006). In a separate study on that collection, stems (MAP 0.276) outperformed raw words (0.225) and character 4-grams (0.244). Adding learned morphological variants to stems, to capture broken plurals and shifting name endings, raised MAP to 0.296 (Darwish & Ali 2012).
Root and light stemmers for Arabic
Root extraction is often template-driven. The Enhanced Stemmer for Arabic Information Retrieval (ESAIR) segments a word in two layers. It matches the core against a pattern of equal length and deletes the pattern letters to obtain the root. It reports about 96% correct roots on a newspaper corpus (Bessou & Touahria 2014). Ambiguity remains the central difficulty, because one surface word can point back to several roots. The Context-Based Arabic Stemmer (CBAS) first lists every candidate root. It then selects one using a co-occurrence model from distributional semantics and reaches 81.5% accuracy on annotated articles (El-Defrawy et al. 2015).
Some stemmers work without any lexicon. A letter-weighting stemmer was used to reduce dictionary glosses to roots for Arabic word sense disambiguation (Merhbene et al. 2014). For noisy tweets, a stemmer with only a small affix grammar picks the shortest permitted stem and scores 77.91% overall (Albogamy & Ramsay 2016). Throughput can matter too. A pipelined field-programmable gate array (FPGA) implementation of a linguistic root extractor processes about 2.08 million words per second (Damaj et al. 2017). Some curated resources, such as the Murabaa lexicons of roots, patterns and clitics, are published separately from the tools that consume them (Bouzoubaa et al. 2026).
Beyond search
Stemming also helps language modeling across language varieties. When both training and test data are stemmed, the perplexity of a Modern Standard Arabic model on Egyptian colloquial speech falls from 12874.2 to 6260.7, roughly half. A small model trained on the dialect itself still does far better, at 184.8 (Rozovskaya et al. 2006).
In dependency parsing, errors in the affix splitting that stemming depends on weigh heavily. One fully automatic Arabic pipeline uses the same segmenter to produce both tokens and stems. There, a wrong split adds or removes tree nodes. As a result, a tokenizer that is 99.34% accurate still accounts for most of the parsing loss from gold to automatic input (Mohamed 2011).
Several newer systems avoid stemming altogether. Embedding-based topic modeling yields readable topics with no stemming or lemmatization (Angelov 2020). Glyph-image character embeddings for Arabic text classification outperform a support vector machine (SVM) baseline that relied on the Khoja stemmer (Daif et al. 2020). Meteor Universal does without stemmers by deriving its resources from parallel text (Denkowski & Lavie 2014). Subword tokenization in large language models is often said to take over part of stemming's original role.
Open questions
- Light or root? Light stemming competes well in retrieval, but its advantage over full root extraction is unproven. Roots still matter for other Arabic tasks, so the better choice may depend on the task (El-Defrawy et al. 2015). Choosing stems by corpus frequency fixes one specific light-stemming error: stripping ان when those letters belong to the core (Kadri & Nie 2006).
- Context. CBAS indicates that a word's surroundings can resolve competing analyses (El-Defrawy et al. 2015). Most deployed stemmers still look at one word at a time.
- Foreign and transliterated words. Stemmers mishandle out-of-vocabulary names. Missing diacritics can make a foreign word fit an Arabic pattern by accident, which is why detection should come before normalization (Nwesri et al. 2006).
- Dialects and user text. Dialectal Arabic lacks dependable stemmers, so tasks such as negation-aware sentiment analysis have to rely on lexicon rules (Al-Harbi 2020). Open-vocabulary social media text works against dictionary-based analysers (Albogamy & Ramsay 2016).
- Morphology in neural models. It is still unclear whether explicit roots and stems should feed into morphology-aware language models, or whether learned subwords make them unnecessary (Bouzoubaa et al. 2026; Al-Khalifa; & Alam 2025).
Key papers
Related concepts
- Lemmatization — Lemmatization is the task of reducing an inflected or variant word form to its lemma, the canonical dictionary form under which it would be…
- Morphology — Morphology is the branch of linguistics that studies the internal structure of words. It covers how words are built from meaningful units…
- TF-IDF — Term frequency–inverse document frequency (TF-IDF) is a weighting scheme that scores how important a word is to one document in a…
- Inverted index — An inverted index is a data structure that maps each term of a vocabulary to a postings list of the documents containing it, usually with…
More papers
Topics: 🧩 Morphology · 🔍 Information retrieval & search · ✂️ Tokenization & segmentation
Shelves: 🤖 Artificial Intelligence
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/stemming/