🔤 Word form
A word form is one concrete written or spoken shape of a word as it appears in text, such as walks or walked. Each form realises a lexeme under a particular set of grammatical features, while the lemma serves as the dictionary citation form for the whole set.
What it is
When people say that "walk", "walks", "walked" and "walking" are "the same word", they mean the same lexeme: one unit of vocabulary with one core meaning. Each of the four strings is a separate word form. One form is chosen by convention to stand for the whole set in dictionaries. That form is the lemma, and for English verbs it is the bare infinitive "walk".
The distinction belongs to morphology, the study of how words are built. It matters wherever language is processed by machine. A search engine, a tagger or a translator sees only word forms and must decide when two forms are really one word.
The set of all word forms of a lexeme, arranged by grammatical function, is its paradigm. The process that produces the forms is inflection. Inflection differs from derivation, which creates new lexemes, as when "walker" is formed from "walk". English paradigms are small:
| Form | Function |
|---|---|
| walk | base form (infinitive; present tense except 3rd person singular) |
| walks | 3rd person singular present |
| walked | past tense and past participle |
| walking | present participle |
Many languages have far larger paradigms. Here are some of the case forms of the Finnish noun TALO 'house':
| Case | Singular | Plural |
|---|---|---|
| nominative | talo | talot |
| genitive | talon | talojen |
| partitive | taloa | taloja |
| inessive ('in') | talossa | taloissa |
| elative ('out of') | talosta | taloista |
| illative ('into') | taloon | taloihin |
English verbs mostly show four or five distinct forms, and "be" shows eight (Cotterell et al. 2019). A Finnish noun has more than a dozen case slots in each number.
The key idea
A word form can be described as a pair: a lemma and a bundle of morphological features. "walked" is WALK plus Tense=Past. "talossa" is TALO plus Case=Ine and Number=Sing.
Universal Dependencies (UD) stores exactly this decomposition. It uses CoNLL-U files, a tab-separated plain-text format with one word per line and one column per annotation layer. The columns include:
- FORM, the word form;
- LEMMA, the lemma;
- UPOS, the universal part-of-speech tag;
- FEATS, the morphological features as attribute–value pairs separated by bars, for example
Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin.
A parallel effort, the UniMorph schema, defines inflectional meanings independently of any one language. It has 23 dimensions, such as case, aspect, evidentiality and politeness, and 212 features (Sylak-Glassman et al. 2015).
Seen this way, two computational problems mirror each other:
- Analysis maps a word form to its lemma and features. This is the task behind lemmatization and morphological tagging.
- Inflection generation maps a lemma and features to a word form.
Stemming is a cruder cousin of analysis. It strips endings without claiming to recover a real lemma.
The difference between forms and lemmas also changes how a corpus is counted. If you count distinct word forms, every inflected variant is a separate type. If you count lemmas, all the variants are grouped together. The choice affects the type–token ratio, the long tail described by Zipf's law, and the number of unseen words a system meets.
The size of a paradigm is not the whole story, though. Cotterell and colleagues distinguish two kinds of complexity:
- Enumerative complexity is the number of distinct forms a paradigm has.
- Integrative complexity is how hard those forms are to predict.
They define integrative complexity as a conditional entropy:
H(M | L) = − Σ_{m,ℓ} p(m, ℓ) · log p(m | ℓ)
Here L is the random variable ranging over lemmas ℓ, and M is the random variable ranging over complete paradigms m. The probability p is the one a fitted model assigns. Low entropy means that once you know the lemma, the remaining forms follow with little surprise.
They measured written noun and verb paradigms in 36 languages. None of the languages in that sample combined very large paradigms with high irregularity, which points to a trade-off between the two. The authors also caution against equating rich morphology with a high rate of out-of-vocabulary words. Predictable paradigms, as in Hungarian, may be easier to process than irregular ones, as in German (Cotterell et al. 2019).
History and founding papers
Two-level morphology and finite-state machines
Two-level morphology took shape in Kimmo Koskenniemi's 1983 doctoral dissertation. In 1984 he presented it at the International Conference on Computational Linguistics (COLING). That paper set out a general computational model in which one formalism serves both recognition and production of word forms (Koskenniemi 1984).
Two-level morphology relates a lexical string directly to a surface string. The lexical string is made of stems and affixes with boundary marks. Constraints apply in parallel, and each constraint is a finite-state transducer. A rule of the textbook shape
a:b ⇒ LC _ RC
says that lexical symbol a may surface as b only between a left context LC and a right context RC. Because transducers run in both directions, the same grammar analyses and generates.
Large finite-state analysers followed this line. SMOR for German covers inflection, derivation and compounding in one transducer. Feature filters control how affixes combine, and morphophonological rules then turn the result into spelling (Schmid et al. 2004).
SMOR was tested on 900 unseen word forms from the ELNC newspaper corpus, 300 from each of three frequency bands. It analysed 767 of them (85%), and only 31 of the analyses it returned were wrong, so roughly 96% were right. A separate count covered every purely alphabetic type in ELNC, and SMOR analysed 66.8% of them (Schmid et al. 2004).
Weighted transducers later made phonological mappings learnable. Cotterell, Peng and Eisner inferred latent underlying morphs together with a stochastic phonology. Their learner predicted held-out surface forms across 7 languages (Cotterell et al. 2015).
Paradigms from data
Statistical work turned word-form generation into supervised learning over whole tables. Durrett and DeNero learned complete paradigms from Wiktionary inflection tables (Durrett & DeNero 2013).
Sylak-Glassman and colleagues then wrote a general parser for Wiktionary tables and mapped its output to their universal schema. This yielded 883,965 paradigms in 352 languages. With that data, Durrett and DeNero's learner could handle 123 language–part-of-speech pairs instead of the original 5 (Sylak-Glassman et al. 2015). Kirov and colleagues later pushed the parsing and normalisation of Wiktionary paradigms to very large scale (Kirov et al. 2016).
Neural inflection and the SIGMORPHON tasks
The SIGMORPHON 2016 shared task framed morphological reinflection as a common benchmark. Systems had to produce a target word form from a lemma, or from another form, plus target features (Cotterell et al. 2016).
Character-level sequence-to-sequence models soon dominated (Faruqui et al. 2016). A single encoder–decoder with explicit feature symbols handled all reinflection directions (Kann & Schütze 2016). Hard monotonic attention lets the decoder either write a character or step to the next input position. It did especially well on small training sets (Aharoni & Goldberg 2017).
The task series kept widening:
- 2017: universal reinflection in 52 languages, plus paradigm completion (Ryan & Cotterell 2017)
- 2018: universal morphological reinflection, co-organised with the Conference on Computational Natural Language Learning (CoNLL) (Cotterell et al. 2018)
- 2019: analysing word forms within their sentence context, and transferring inflection knowledge between languages (McCarthy et al. 2019)
- 2020: typologically diverse languages (Vylomova et al. 2020)
- 2021: generalisation across languages (Pimentel et al. 2021)
In 2017, the low-data track offered 100 training examples per language. In that setting, an unmodified neural model of the Kann and Schütze type reached only about 0–1% accuracy for most languages. Synthetic examples, made by replacing the shared stem with random characters, made neural reinflection usable (Silfverberg et al. 2017).
In 2019, hallucinated data combined with transfer from related languages raised macro-averaged accuracy by 15 points over the earlier best system. It placed first in the task (Anastasopoulos & Neubig 2019).
How it is used today
Inventories of word forms are core resources for lemmatization, spell checking and lexicography. MorphyNet extracts inflectional and derivational data from Wiktionary for 15 languages. It holds about 10.1 million inflectional entries, and manual evaluation puts its precision near 99% (Batsuren et al. 2021).
Morphological features also feed downstream analysis. In Arabic dependency parsing, several predicted features helped: person, number, gender, definiteness and the undiacritized lemma. Functional rather than form-based number and gender helped further, because they handle broken plurals correctly. Case, by contrast, failed once it had to be predicted (Marton et al. 2013).
Probing studies find that subject and object number and main-clause tense remain decodable from sentence encoders even when these features are useless for the task. Recoverable inflectional information is therefore not proof that a model relies on it (Ravichander et al. 2021).
Word forms also matter for robustness and fairness. MORPHEUS replaced words with other inflected forms of the same lemma. This seriously degraded question answering and English–French machine translation. Fine-tuning on such perturbations reduced the damage (Tan et al. 2020). A related encoding splits words into base and inflection to better serve non-standard varieties of English (Tan et al. 2020).
For Arabic, gender reinflection rewrites first-person sentences into the other gender's word forms. It works as a post-processing layer around translation (Habash et al. 2019).
Open questions
- How should a word form be defined where spelling is a poor guide, for instance in languages that are mainly spoken? Complexity estimates computed on written paradigms may differ from estimates based on speech (Cotterell et al. 2019).
- Can models generalise to lemmas and feature combinations never seen in training? Accuracy in the low-data setting tracked how many test feature sets were attested during training (Silfverberg et al. 2017). Generalisation across languages is still being probed (Pimentel et al. 2021).
- How should transfer be used for low-resource languages? In one set of experiments, transfer worked best from genetically close languages that share a script. Hallucination alone came within a few points of transfer plus hallucination, which makes it a workable option for language isolates (Anastasopoulos & Neubig 2019).
- Can the different feature inventories be reconciled? The universal schema, UD FEATS and language-specific tagsets still differ (Sylak-Glassman et al. 2015), and resources overlap only partly (Batsuren et al. 2021).
Key papers
Related concepts
- Lemma — A lemma is the citation form of a word: the single form, such as a dictionary headword, chosen to stand for every inflected form of one…
- Morphology — Morphology is the branch of linguistics that studies the internal structure of words. It covers how words are built from meaningful units…
- Lemmatization — Lemmatization is the task of reducing an inflected or variant word form to its lemma, the canonical dictionary form under which it would be…
- Type–token ratio — Type–token ratio (TTR) is a measure of lexical diversity: the number of distinct word forms (types) in a text divided by the total number…
More papers
Topics: 🧩 Morphology
Shelves: 🗣️ Language and Linguistics
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/word-form/