🔠 Morphology

About 17 minutes to read · every paper named here has its own page in this library

What it is

Morphology asks what a word is made of. The English word unhappiness has three pieces:

• the prefix un-, which means "not";
• the root happy, which carries the core meaning;
• the suffix -ness, which turns an adjective into a noun.

Each of these minimal units of form and meaning is a morpheme. Some morphemes can stand alone as words (happy, cat) and are called free. Others, such as -ness or the plural -s, must attach to something and are called bound. A root is the morpheme that carries the main lexical meaning. An affix is a bound morpheme added before the root (prefix), after it (suffix), around it (circumfix) or inside it (infix).

Linguists usually separate three kinds of word-building:

• Inflection changes the form of a word to fit its grammatical context without making a new word. Walk, walks, walked and walking are all forms of one lexeme, WALK, whose citation form is called the lemma. The full set of forms of a lexeme is its paradigm.
• Derivation creates a new lexeme, often of a different word class: happy → happiness, teach → teacher, nation → national → nationalize.
• Compounding joins two or more roots into one word: blackboard, toothbrush, or German Donaudampfschiff ("Danube steamship").

English has little inflection, and a regular verb has only four forms. Many languages have far more. A Turkish word can stack a long chain of suffixes on one root, and those suffixes can move the word between noun, verb and adverb categories. Vowel harmony then changes how each suffix is spelled.

A single Turkish surface word can also allow several readings. The form alınmış can be built from "red", from "forehead", or from a passive of "take", among other analyses (Oflazer 1993). Arabic shows a different pattern. Its words are usually written without short vowels, and the same written form often has several possible analyses (Habash & Rambow 2005).

Language types

The classic morphological typology sorts languages by how they package grammatical meaning into words. It goes back to the Schlegel brothers and Humboldt in the nineteenth century and was refined by Sapir (1921). Its original types are isolating, agglutinative and fusional. Humboldt added an incorporating type, now usually called polysynthetic. The templatic type is a later addition, often listed separately because Semitic word structure fits none of the original classes well. Real languages mix these types, so the categories describe tendencies, not boxes.

The templatic type is the hardest to fit into a "string of pieces" picture. In Arabic, the root k-t-b is linked to writing. Different patterns of vowels and extra consonants turn it into kataba "he wrote", kātib "writer", maktab "office" and kitāb "book". The root never surfaces on its own, and no contiguous substring of the word corresponds to it. That is why Semitic languages have long been a test case for computational models of morphology.

Morphology sits between two other fields of grammar:

• Syntax arranges words into sentences.
• Phonology governs how morphemes are pronounced when they meet.

It also feeds lexicography, which decides what counts as a dictionary entry. It feeds psycholinguistics as well, which studies how speakers store and process complex words. Derivational databases such as MorphoLex, built for about 70,000 English words, serve that line of research (Sánchez-Gutiérrez et al. 2018).

How it works

A morphological description has two parts:

• Morphotactics states which morphemes may combine and in what order.
• Morphophonology states how their shapes change when they meet. English -s surfaces differently in cats, dogs and horses, and a Turkish suffix vowel harmonizes with the stem.

In computational linguistics, such grammars are turned into programs that map in two directions:

• Analysis maps a surface form to a lemma plus a bundle of features, such as walked → WALK + Verb + Past.
• Generation maps a lemma and feature bundle to the surface form.

Ambiguity and disambiguation

An analyzer that looks at a word in isolation usually returns several candidate analyses. Choosing the right one in context is morphological disambiguation, a close relative of part-of-speech tagging. In richly inflected languages the tag is no longer a few dozen labels. It becomes a combination of many features, and the space of possible tags explodes:

• For Arabic, about 333,000 fully specified tags are theoretically possible. Only about 2,200 are attested in the opening 280,000 words of the Penn Arabic Treebank (Habash & Rambow 2005).
• Across six languages annotated on the same novel, tagset sizes ranged from 139 tags for English to 970 for Czech and 1,033 for Slovene (Hajic 2000).

Sparse data of this kind is a central practical problem of the field.

Why it matters for applications

Rich morphology multiplies the number of distinct word types. In a seven-language parallel corpus, the average number of tokens per word type fell from 12.1 in English to 4.84 in Hungarian. Rare words are therefore far more common in Hungarian (Clark 2003).

Any system that treats each surface form as an atomic symbol then sees many forms too rarely to learn them. This affects machine translation, word embeddings, dependency parsing and language models. The methods below are largely different answers to this problem.

History

Structural linguists in the mid-twentieth century tried to discover morphemes mechanically from distributional evidence. Harris (1955) proposed counting how many different letters can follow a given prefix. Hammarström and Borin's survey traces unsupervised morphology learning back to these structuralist discovery procedures (Hammarström & Borin 2011).

The computational era began with finite-state methods. Koskenniemi's two-level morphology (1983) expressed morphophonological rules as constraints between a lexical and a surface level, compiled into finite-state transducers. Kemal Oflazer's description of Turkish in this framework rested on a lexicon of roughly 23,000 roots. It handled nearly every exception to its phonological and morphotactic rules and served as the morphological front end for several Turkish parsers (Oflazer 1993). A freely available wide-coverage analyzer for English appeared in the same period (Karp et al. 1992).

Around 2000 the field shifted toward data. Jan Hajič ran experiments on parallel annotated text in several richly inflected languages. A simple tagger with a morphological dictionary did as well as or better than a model trained on about 100,000 annotated tokens without one. He concluded that, for such languages, building dictionaries can pay off more than annotating more text (Hajic 2000). At the same time, Goldsmith (2001) and Creutz and Lagus (2002) put unsupervised segmentation on a statistical footing (Goldsmith 2001; Creutz & Lagus 2002).

From the mid-2000s, discriminative statistical models took over disambiguation. Conditional random fields (CRFs) were applied to Japanese analysis (Kudo et al. 2004), and support vector machines to Arabic (Habash & Rambow 2005). Morphology was also used to improve statistical machine translation (Lee 2004; Goldwater & McClosky 2005).

The 2010s brought neural models and large multilingual resources. The SIGMORPHON shared tasks on reinflection ran in 2016, 2017 and 2018 (Cotterell et al. 2016; Ryan & Cotterell 2017; Cotterell et al. 2018). They were built on the UniMorph inventory of paradigms (Kirov et al. 2018). In the same years, probing studies began to ask what neural networks learn about word structure on their own (Belinkov et al. 2017).

Methods

Finite-state analyzers

A finite-state transducer (FST) reads one string and writes another, so a single machine can serve as both analyzer and generator. Lexicon, morphotactics and rules are compiled into one transducer that is fast and reversible.

SMOR is a German analyzer built with the open SFST tools. It handles productive derivation, compounding and inflection in one system. On 900 unseen word forms it analyzed 85%, and roughly 96% of the analyses it returned were correct. It runs at more than 4,000 words per second (Schmid et al. 2004).

For Semitic root-and-pattern morphology, a single input tape is awkward. MAGEAD built on Kiraz's multitape approach, with separate tapes for root, pattern and other components. It added a tape that keeps phonology apart from orthography and is reported as the first wide-coverage implementation of this kind. Adapted to Levantine Arabic, it recovered about 95% of contextually correct analyses. A Modern Standard Arabic (MSA) baseline managed roughly 53–60% (Habash & Rambow 2006).

Dialect-specific lexicons matter. The Egyptian Arabic analyzer CALIMA contained the correct analysis for about 84% of test words, and about 92% when combined with an extended MSA analyzer (Habash et al. 2012).

Large lexicons also need compact encodings. The Czech MorphoDiTa dictionary covers 120 million form–tag pairs from about 1 million lemmas and 3,922 tags. It is compressed into a binary of about 2 MB by grouping shared endings into templates (Straková et al. 2014).

For French, the Lefff lexicon compiles a compact intensional description into a full list of inflected forms. Adding it to a part-of-speech (POS) tagger raised accuracy on unknown words from 86.1% to 90.1% (Sagot 2010).

Stemming and lemmatization

Stemming cuts words down to a common truncated form, usually with suffix-stripping rules such as Porter's (1980) algorithm. It is crude but useful for search. Lemmatization instead returns the dictionary form, which requires knowing the word class and often the context. The two can differ sharply:

• A stemmer turns "studies" into "studi", while a lemmatizer returns "study".
• A stemmer leaves "better" untouched, whereas a lemmatizer maps it to "good".

Data-driven lemmatizers treat the task as classification over edit operations. Morfette predicts the shortest edit script that turns the reversed word form into the reversed lemma. A beam search combines this with a morphological tagger. Joint tag-and-lemma accuracy was about 96% for Romanian and about 81–84% for Polish. In the Polish IPI PAN corpus, only 44.44% of tokens are identical to their lemma (Chrupala et al. 2008).

LEMMING generates candidate lemmata with edit trees. It scores them with a log-linear model and trains jointly with a higher-order CRF tagger (Müller et al. 2015; Mueller et al. 2013). For Czech it cut lemmatization error from 4.05% to 1.58% (Müller et al. 2015).

Analyze-then-disambiguate pipelines

A long Arabic tradition runs an analyzer first and then lets classifiers choose among its outputs. Habash and Rambow trained per-feature classifiers for POS, clitics, gender, number, person, voice and aspect. They then picked the analysis that agreed best with the predictions, reaching 97.6% POS accuracy on one test set (Habash & Rambow 2005). Later systems extended this design:

• MADA added lexeme language models and per-task weight tuning. Lemmatization accuracy rose from 85.5% to 96.3% over an untuned baseline (Roth et al. 2008).
• Swapping in an Egyptian analyzer and dialect treebank produced MADA-ARZ. On full analysis it scored 75.4%, against 27.0% for the MSA system (Habash et al. 2013).
• MADAMIRA reimplemented the pipeline in Java with near-identical accuracy, for example 95.9% POS accuracy on MSA. It runs more than an order of magnitude faster (Pasha et al. 2014).

For unsegmented scripts, segmentation and tagging must be solved together. Kudo, Yamamoto and Matsumoto defined CRFs over a lexicon-derived word lattice, so that paths with different numbers of tokens compete. On the Kyoto corpus, segmentation F-score reached 98.96 (Kudo et al. 2004).

A later pointwise approach decides each character boundary independently with logistic regression (Neubig et al. 2011):

• In-domain, it beat the CRF-based MeCab (98.03% vs 97.31% word/POS F-measure).
• It trained about three times faster than a comparable CRF.
• It allowed training on partially annotated text.

Multilingual pipelines such as UDPipe learn tokenization, lemmas, features and dependency trees directly from Universal Dependencies (UD) treebanks. Its first release covered 32 of the 37 treebanks of UD 1.2 (Straka et al. 2016).

Unsupervised segmentation: Morfessor and its relatives

Unsupervised methods split words into morph-like units using raw text alone. Morfessor, introduced by Creutz and Lagus (Creutz & Lagus 2002), is the best-known example. Its Baseline model is trained by maximum a posteriori estimation, which is closely tied to minimum description length (MDL):

θ* = argmin_θ [ −log P(θ) − log P(D | θ) ]

Here θ is a lexicon of morphs together with their probabilities, and D is the training corpus of words. P(θ) is a prior that rewards a small lexicon of short units. P(D | θ) is the likelihood of the corpus when each word is segmented into morphs from that lexicon. The best lexicon θ* balances a compact inventory against a good fit to the data.

Training uses a recursive binary-splitting search, and new words are segmented with Viterbi decoding. Morfessor 2.0 reimplemented this as an open Python toolkit. It added semi-supervised training, in which weighting the corpus likelihood trades over-segmentation against under-segmentation (Smit et al. 2014).

Later models enriched this picture:

• A log-linear model with MDL-like priors on lexicon size and on morphemes per word lowered the F1 error of Morfessor Categories-MAP by 11% (relative) on the Arabic treebank (Poon et al. 2009).
• A Bayesian model that segments two languages at once from parallel Bible phrases lowered relative error by up to 24% compared with its monolingual version (Snyder & Barzilay 2008).

The survey by Hammarström and Borin found that such systems reach high accuracy mainly for languages with simple single-slot concatenative morphology. Assigning meanings to affixes remained largely unsolved (Hammarström & Borin 2011).

Neural inflection and reinflection

Morphological inflection generation takes a lemma and a feature bundle and produces the inflected form. Reinflection does the same starting from another inflected form. Earlier, Durrett and DeNero had learned, with supervision, to predict complete inflection tables from a lemma (Durrett & DeNero 2013). The single-form task then became a standard benchmark through SIGMORPHON (Cotterell et al. 2016).

The winning 2016 approach was a single character-level encoder–decoder that receives the morphological features as part of its input (Kann & Schütze 2016; Silfverberg et al. 2017). Aharoni and Goldberg replaced soft attention with hard, nearly monotonic attention. At each step the decoder either writes a character or advances its pointer over the input. The gains were largest on small training sets (Aharoni & Goldberg 2017).

Low-resource settings motivated data augmentation. One method finds the longest common substring of lemma and form and treats it as a stem. It then replaces that stem with random letters to create synthetic training pairs (Silfverberg et al. 2017).

Combining this kind of "hallucination" with transfer from related languages was tested in the cross-lingual track of SIGMORPHON 2019. It gave a 15-percentage-point gain in macro-averaged accuracy over the prior state of the art. The system generated 10,000 synthetic examples per language whose real training sets could be as small as about 50 (Anastasopoulos & Neubig 2019).

Morphology inside distributed representations

Standard embeddings give every surface form its own vector. Morphological recursive networks build a word vector from its morphemes instead:

p = tanh( W_m [ x_stem ; x_affix ] + b_m )

The terms of the formula are:

• x_stem and x_affix are the vectors of the stem and the affix.
• [ ; ] places them one after the other in a single vector.
• W_m is a learned weight matrix and b_m a bias vector, both shared across all words.
• The tanh function squashes the result into the vector p, which represents the combined unit.

Applied recursively, this builds "unfortunately" from un + fortunate + ly. On the Rare Word (RW) similarity set, the context-sensitive model (csmRNN) raised the Spearman correlation (×100) of the HSMN embeddings from 1.97 to 22.31 (Luong et al. 2013).

Analogy tests reveal limits. The Bigger Analogy Test Set (BATS) is a balanced set of 40 relations that includes inflectional and derivational ones. On it, the best configurations reached only 22.1% average accuracy for a count-based model reduced with singular value decomposition (SVD), and 28.5% for GloVe (Gladkova et al. 2016).

Morphology against subword tokenization

Modern large language models rarely use morphological analysis at all. They split text into pieces with subword tokenization. Two common schemes are byte-pair encoding, adapted for translation by Sennrich et al. (2016), and unigram language-model segmentation by Kudo (2018).

These algorithms choose units by frequency, not by meaning. A frequent word stays whole, a rare word is broken into fragments, and the fragments may or may not coincide with morphemes. Subword units are cheap, language-independent and handle any input string. Their boundaries can still cut across morphemes and miss non-concatenative structure, as in Semitic templates.

Evidence on how much this matters is mixed. One controlled comparison covered 21 Europarl languages, using n-gram models and character-level long short-term memory (LSTM) models. Languages with richer inflection were harder to model for both. The correlation between modeling cost and inflectional complexity was Spearman ρ = 0.59. It vanished (ρ = −0.13) once the text was lemmatized (Cotterell et al. 2018). Character-level models alone do not neutralize morphological richness.

In neural translation, character-aware encoders captured morphology much better than word-level ones. Arabic POS probing accuracy rose from 89.62% to 95.35%, with the largest gains on unseen words (Belinkov et al. 2017). Such probing results need care, though. Properties stay decodable from representations even when they are useless for the task, so a high probe score does not show that a model uses the property (Ravichander et al. 2021).

Explicit morphology can also make systems more robust and fairer:

• An adversary that only swaps inflections of the same lemma degraded question answering and translation models. Adversarial fine-tuning reduced the drop of a Transformer-big translator under attack from 56.25% to 20.20% (Tan et al. 2020).
• In gender-marking languages, flipping a noun's gender requires reinflecting its agreeing words. A morphology-aware augmentation pipeline lowered gender stereotyping in language models by an average factor of 2.5 across four languages (Zmigrod et al. 2019).

These lines of work connect morphology to interpretability and to bias, fairness and safety.

Morphology for translation

Before neural systems, morphological preprocessing was a standard lever in statistical translation. Results are given in BLEU, a translation-quality metric that scores n-gram overlap with reference translations on a 0–1 scale.

One method split Arabic words into prefixes, stems and suffixes and then decided per affix whether to keep, merge or delete it. With 3,500 training sentence pairs, it raised IBM Model 1 BLEU from 0.10 to 0.25 (Lee 2004). For Czech–English, replacing forms with lemmas and turning selected tags into pseudo-words raised test BLEU from 0.270 to 0.333 (Goldwater & McClosky 2005).

In the other direction, a separate inflection generator was given stemmed reference translations. It picked the right target form in 91.6% of cases for Russian and 91.0% for Arabic. A trigram language model managed 81.0% and 69.4% (Toutanova et al. 2008).

Datasets and benchmarks

Evaluation in morphology depends on gold paradigms, annotated corpora and shared tasks. The main resources and what they measure:

The feature schema behind UniMorph was derived from linguistic typology. It has 23 dimensions of meaning, such as case, aspect, evidentiality and politeness, and 212 features. A general parser for Wiktionary inflection tables produced 883,965 paradigms in 352 languages (Sylak-Glassman et al. 2015).

UniMorph 2.0 packaged such data for shared tasks (Kirov et al. 2018). Release 3.0 followed an audit that judged about 25% of sampled entries erroneous. It rebuilt the Wiktionary scraping and grew the resource to 118 languages and 14.8 million triples, up from 52 languages in 2.0 (McCarthy et al. 2020).

The shared tasks themselves moved quickly:

• The 2016 task introduced reinflection (Cotterell et al. 2016).
• The 2017 task covered 52 languages (Ryan & Cotterell 2017). Its training sets had 100, 1,000 and 10,000 examples in the low, medium and high settings (Silfverberg et al. 2017).
• The 2018 edition continued the universal reinflection format (Cotterell et al. 2018).
• The 2019 edition added a cross-lingual transfer track (Anastasopoulos & Neubig 2019).

For segmentation, the Morpho Challenge evaluations supplied gold annotations for Finnish, English, German, Arabic and Turkish. They also included retrieval-based evaluation (Hammarström & Borin 2011).

For embeddings, the Rare Word similarity set (Luong et al. 2013) and the BATS analogy set (Gladkova et al. 2016) test whether vectors capture morphological relations.

Classic tagging comparisons use parallel annotated text, such as the multilingual edition of Orwell's 1984, so that differences reflect the languages rather than the content (Hajic 2000; Clark 2003).

People and institutions

Much of the field's computational work on Arabic comes from a small, long-running collaboration. Nizar Habash, now at New York University Abu Dhabi and New York University, co-authored the following systems:

• the single-pass approach to Arabic tokenization and tagging (Habash & Rambow 2005);
• the MAGEAD multitape analyzer (Habash & Rambow 2006);
• the Egyptian analyzer CALIMA (Habash et al. 2012);
• MADA and its dialect version (Roth et al. 2008; Habash et al. 2013);
• MADAMIRA (Pasha et al. 2014).

Owen Rambow, now at Stony Brook University, was his co-author on several of these systems (Habash & Rambow 2005; Habash & Rambow 2006). Ramy Eskander contributed to the Egyptian Arabic analyzer and to MADAMIRA (Habash et al. 2012; Pasha et al. 2014).

Jan Hajič, at Charles University and the Czech Academy of Sciences, argued for the value of morphological dictionaries in inflection-heavy languages (Hajic 2000). With colleagues in Prague he released the MorphoDiTa and UDPipe tools (Straková et al. 2014; Straka et al. 2016).

The multilingual turn is tied to Ryan Cotterell, now at ETH Zurich. He co-organized the SIGMORPHON reinflection tasks (Cotterell et al. 2016; Cotterell et al. 2018). He studied how morphology affects language modeling (Cotterell et al. 2018) and co-developed LEMMING (Müller et al. 2015).

Christo Kirov, now at Google, and David Yarowsky, at Johns Hopkins University, co-designed the universal feature schema and Wiktionary extraction behind UniMorph (Sylak-Glassman et al. 2015; Kirov et al. 2018). Mans Hulden co-organized the first reinflection task (Cotterell et al. 2016).

Other contributions span neural methods and analysis. Yoav Goldberg, at Bar-Ilan University, co-introduced hard monotonic attention for inflection (Aharoni & Goldberg 2017). Yonatan Belinkov, now at the Technion, led the probing study of morphology in neural translation and later questioned what probing can show (Belinkov et al. 2017; Ravichander et al. 2021). Josef van Genabith, at the German Research Centre for Artificial Intelligence and Saarland University, co-developed Morfette (Chrupala et al. 2008).

Open questions

Several problems remain open:

• Do language models need morphology? Subword tokenizers are morphology-blind, yet large models perform well in many languages. It is still unresolved whether richly inflected languages carry more information, or whether current models simply fail to exploit predictable morphemes (Cotterell et al. 2018). Probing alone cannot settle the matter, because a decodable property need not be a used one (Ravichander et al. 2021).
• Non-concatenative structure. Root-and-pattern words, stem changes and reduplication fit poorly into both segmentation models and frequency-based subword vocabularies. Unsupervised learners succeed mainly on simple concatenative morphology (Hammarström & Borin 2011). Explicit multitape treatments exist but require hand-built resources (Habash & Rambow 2006).
• From form to meaning. Finding segments is easier than labeling what each affix means, a step that remains essentially unsolved without supervision (Hammarström & Borin 2011).
• Data quality and coverage. Crowd-sourced paradigm data contain systematic errors (McCarthy et al. 2020). Many low-resource languages still have only a few dozen annotated forms (Anastasopoulos & Neubig 2019). Augmentation helps, but it fails when lemma and form share little material, as with Navajo in one shared-task system (Silfverberg et al. 2017).
• Dialects and unstandardized spelling. Tools for standard varieties transfer poorly to spoken dialects, which need their own lexicons and annotation (Habash et al. 2012; Habash et al. 2013).
• Derivation and compounding. Shared tasks have focused on inflection, while productive derivation and ad hoc compounds stay harder. Even a dedicated German analyzer covered only part of the low-frequency forms it was tested on (Schmid et al. 2004). Embeddings likewise handle derivational relations poorly (Gladkova et al. 2016).
• Robustness and fairness. Models trained on standard inflection patterns break under synthetic inflectional perturbations modeled on the variation in learner and dialect English (Tan et al. 2020). Debiasing methods must respect agreement morphology (Zmigrod et al. 2019).

For further reading, see the topic on tokenization and the topic on multilingual natural language processing.

The library in figures