📗 Lemmatization
Lemmatization is the task of reducing an inflected or variant word form to its lemma, the canonical dictionary form under which it would be listed. The choice of lemma takes into account the word's part of speech and its context in the sentence.
What it is
A text contains word forms, but a dictionary lists lemmas. Lemmatization connects the two. It maps better to good, was to be, and mice to mouse.
In a richly inflected language a single lemma can stand behind dozens of forms. A Czech or Latin noun changes for case and number, and a verb changes for person, tense and mood. Collapsing these forms makes counting, searching and comparing words practical in corpus linguistics, lexicography and information retrieval.
Lemmatization is not the same as stemming. A stemmer cuts off endings by rule and can return a string that is not a word. A lemmatizer must return a real headword, and often it has to know the part of speech first.
The English form saw is the noun saw in one sentence and the past tense of see in another. The Latvian form ceļu can come from a verb or from either of two nouns, and only context decides which (Bergmanis & Goldwater 2018).
The key idea
The task has two parts: generating candidate lemmas and choosing among them. A form may admit several analyses, and morphological tags and the lemma constrain each other. Tags often settle which lemma is correct. Conversely, a known lemma can disambiguate syncretic forms, where one spelling is shared by several feature combinations. An example is the gender of a German plural noun (Müller et al. 2015).
Systems represent the form-to-lemma mapping in one of three main ways:
| Approach | How a lemma is produced | Typical strength |
|---|---|---|
| Dictionary / analyzer | look up the form in a lexicon of paradigms, then disambiguate | precise on known words |
| Edit-rule classification | predict a transformation (an edit tree or rule class) and apply it | compact, generalizes to unseen forms |
| Character sequence-to-sequence | generate the lemma letter by letter | needs only lemmatized text |
The middle row reframes lemmatization as part-of-speech tagging-like classification. Each token receives a label such as "remove -s, add nothing", and that label is executed deterministically. Recent work on Arabic across several genres still uses this framing (Saeed & Habash 2025).
Systems are usually scored by exact-match accuracy over tokens:
Acc = (1/N) · Σᵢ 𝟙[ l̂ᵢ = lᵢ ]
Here N is the number of evaluated tokens, l̂ᵢ is the predicted lemma for token i, and lᵢ is the gold lemma. The indicator 𝟙 equals 1 when the two strings are identical and 0 otherwise.
Results are commonly reported separately for ambiguous tokens and for tokens unseen in training, because these are the hard cases (Bergmanis & Goldwater 2018).
History and founding papers
Dictionaries, rules and finite-state morphology
The oldest lemmatizers were lexicons with lookup. Classic finite-state morphology is associated with Koskenniemi (1983) and with Beesley and Karttunen (2003). It compiled lexicons and rules into transducers that relate surface forms to lemma-plus-tag analyses. Lemmatization then becomes analysis followed by disambiguation.
Lexicons remain central for languages with large paradigms. The Czech toolkit MorphoDiTa compresses the MorfFlex dictionary into shared ending templates. MorfFlex holds about 120 million form–tag pairs drawn from roughly 1 million lemmas. The toolkit reaches 97.80% lemma accuracy on the Prague Dependency Treebank 2.5 (Straková et al. 2014).
Lexicons can also be extended by rule. A finite-state guesser constrained by contextual tags lemmatized unknown Arabic words and produced a free lexicon of 18,399 entries (Attia et al. 2012).
Statistical disambiguation
In Arabic, short vowels are normally omitted, which multiplies the possible readings of a written word. A 2008 extension of the MADA system ranked candidate analyses from the Buckwalter Arabic Morphological Analyzer (BAMA). It used support vector machines (SVMs) trained separately for each morphological feature, together with lexeme language models. The same work introduced lemma choice as a separate task for Arabic.
On a held-out test half of the ATB3 development set, tuned lexemic features raised lemma-choice accuracy to 96.3%. The untuned baseline, which used only the 14 per-feature predictions, reached 85.5% (Roth et al. 2008).
A different line of work skips the dictionary altogether. It predicts a root-and-pattern template and fills in the root consonants by rule (Attia et al. 2016).
LEMMING (2015) generated candidate lemmas from edit trees, which record which prefix and suffix strings to delete and insert. It scored the candidates with a log-linear model and could also be trained jointly with a conditional random field (CRF) tagger. For Czech it cut lemmatization error by about 60%, from 4.05% to 1.58% (Müller et al. 2015).
Neural lemmatizers
Lematus (2018) adapted an attentional encoder-decoder to write lemmas character by character. Its input is the target word plus a window of surrounding characters, and it needs no part-of-speech (POS) tags. Across 20 Universal Dependencies (UD) treebanks, this context raised accuracy on ambiguous words from 88.8% to 92.4% (Bergmanis & Goldwater 2018).
Later work followed several directions:
- A single multi-task recurrent network was trained for both tagging and lemmatization (Arakelyan et al. 2018).
- Joint contextual models combined tagging and generation (Malaviya et al. 2019).
- Cheap subword signals such as FastText n-grams were added to Arabic sequence-to-sequence (seq2seq) lemmatizers. They reached up to a 22% relative error reduction on Modern Standard Arabic (Zalmout & Habash 2020).
Studies of Slovenian, Croatian and Serbian examined what the neural shift actually improved (Ljubešić & Dobrovoljc 2019; Stankovic et al. 2020).
How it is used today
Lemmas in Universal Dependencies
In Universal Dependencies, every token in a CoNLL-U file carries a LEMMA column next to its form, POS tag and morphological features. UDPipe became a widely used trainable pipeline for this format. It runs from raw text through tokenization, tagging and lemmatization to dependency parsing. It was trained for all UD 2.0 treebanks and served as the official baseline of the CoNLL 2017 shared task (Straka & Straková 2017).
Later systems keep the edit-rule idea with stronger encoders. A lemmatiser for Bokmål and Nynorsk Norwegian follows the 2019 UDPipe approach, fine-tuning BERT models to assign each token an edit-rule class. It pairs this with a full-paradigm lexicon. Raw string matching barely separated the configurations. Manual analysis, however, showed that the lexicon removed more than half of the genuine errors (Yildirim et al. 2026).
Similar UD-oriented pipelines now cover smaller languages such as West Frisian (Heeringa et al. 2022) and Occitan (Miletić 2023).
Pretrained and large models
Multilingual encoder-decoder models can be fine-tuned as text-to-text lemmatizers. For Nepali, mT5-base reached 96.1% accuracy using a set of 8,000 word–lemma pairs split 80/20 into training and test data. Lemmatization also improved downstream retrieval (Regmi et al. 2026).
For dialectal Arabic, seq2seq models combined with existing taggers improved average lemma accuracy by roughly 12 points compared with taggers built for each dialect (Saeed & Habash 2025).
Large language models also appear as post-correctors. On Justinian's Digest, GPT-4o-mini was prompted in a few-shot learning setup on top of LatinPipe, and this gave the best results. Its gains were concentrated on frequent lemmas and verb forms (Marongiu & Sassolini 2026).
Historical languages
Lemmatization is the entry point for annotating historical corpora, where spelling often varies and a written standard may be lacking. Two pilot studies on Old English built lemma inventories by hand on top of the Nerthus lexical database. One covered class 2 weak verbs and identified 1,064 lemmas (Sáenz 2015). The other covered superlative adverbs (Bghiyel 2021).
Other historical work includes:
- enriching Middle English parsed corpora with verb lemmas (Trips & Percillier 2020);
- evaluating lemmatizers for medieval French (Holgado et al. 2021);
- assembling benchmarks for Old Church Slavonic (Nawaz et al. 2026).
For Latin, corpora follow conflicting lemmatization conventions. The LiLa knowledge base links them through a shared collection of lemmas (Mambrini & Passarotti 2019). It was later used to reconcile divergent treatments of participles (Passarotti et al. 2025). A comparable lemma bank now exists for Ancient Greek (Swaelens et al. 2026).
Open questions
What counts as the right lemma is partly a convention, not a fact about the language. Several cases show this:
- Should a participle be lemmatized under its verb or under an adjective? Latin resources disagree (Passarotti et al. 2025).
- Should a Nepali verb map to a bare stem or to its dictionary infinitive (Regmi et al. 2026)?
- How should one headword be chosen when dictionaries themselves differ? Wiktionary editions show how lemmatization policy interacts with etymology and with how much information a page carries (Verdizade 2021).
Evaluation is also fragile. Exact match punishes valid spelling variants in Norwegian (Yildirim et al. 2026). In dialectal Arabic, roughly 23% of apparent errors came from faulty gold lemmas (Saeed & Habash 2025).
Gold data for less-resourced varieties is often built cheaply. One approach accepts the lemma automatically where two lemmatizers agree and sends only the conflicts to human annotators (Belayachi & Mazroui 2026). That approach saves effort but can leave errors that both tools share.
Unseen words remain a major weakness. In the Lematus study, languages with a higher share of unseen words had lower accuracy, with a rank correlation of −0.73 across languages (Bergmanis & Goldwater 2018). Neural generators can also hallucinate non-existent lemmas (Saeed & Habash 2025).
It is still unsettled whether lexicons, edit-rule constraints or pretrained models best curb these failures. The question matters most for low-resource languages and historical languages. Recent comparisons favour hybrids that combine a learned model with a lexicon filter (Yildirim et al. 2026) or with post-correction by a large language model (Marongiu & Sassolini 2026).
Key papers
Related concepts
- Lemma — A lemma is the citation form of a word: the single form, such as a dictionary headword, chosen to stand for every inflected form of one…
- Stemming — Stemming is the reduction of inflected or derived word forms to a common stem, usually by stripping affixes with rules or statistics, so…
- Morphology — Morphology is the branch of linguistics that studies the internal structure of words. It covers how words are built from meaningful units…
- Part-of-speech tagging — Part-of-speech tagging is the task of assigning each token in a text a grammatical category, such as noun, verb, adjective or determiner…
More papers
Topics: 🧩 Morphology · 🔤 Part-of-speech tagging · 🌐 Universal Dependencies
Shelves: 🗣️ Language and Linguistics · 🤖 Artificial Intelligence
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/lemmatization/