🔤 Part-of-speech tagging
Part-of-speech tagging is the task of assigning each token in a text a grammatical category, such as noun, verb, adjective or determiner, from a fixed tagset. The tag is chosen from the word itself and its surrounding context, which resolves words that can belong to several categories.
What it is
Many word forms are ambiguous between categories. In "The can rusted", can is a noun, not a modal verb, and a tagger must work this out from context (Brill 1995).
A POS tagger reads a tokenised sentence and outputs one label per token. Parsers, lemmatisers and named-entity recognisers have traditionally taken these labels as input. So have shallow-analysis components, such as recognising phrases, disambiguating noun senses and labelling grammatical functions (Cutting et al. 1992).
What counts as a "tag" depends on the tagset, and tagsets vary widely in size:
- In the Brown Corpus, 192 distinct tags occur, but only 96 of them appear more than 100 times (Brill 1992).
- The Penn Treebank Wall Street Journal (WSJ) data uses 45 tags (Plank et al. 2016).
- Coarse cross-lingual inventories are much smaller. One set has twelve universal categories (Petrov et al. 2012; Das & Petrov 2011), and Universal Dependencies uses 17 (Plank et al. 2016).
Morphologically rich languages push in the opposite direction. German Tiger tags number over 700, and Czech Academic Corpus tags about 1200 (Schmid & Laws 2008). For Arabic, roughly 333,000 fully specified tags are theoretically possible. Only about 2,200 of them occur in the first 280,000 words of text of the Penn Arabic Treebank (Habash & Rambow 2005). In such languages, tagging merges with full morphological disambiguation.
The key idea
A tag depends both on the word and on the tags of its neighbours. The classic generative formulation is a hidden Markov model (HMM). It scores a whole tag sequence and picks the best one:
t̂₁…t̂ₙ = argmax over t₁…tₙ of ∏ᵢ P(wᵢ | tᵢ) · P(tᵢ | tᵢ₋₂, tᵢ₋₁)
Here w₁…wₙ are the words of the sentence and t₁…tₙ is a candidate sequence of tags. P(wᵢ | tᵢ) is the emission probability: how likely the word wᵢ is given the tag tᵢ. P(tᵢ | tᵢ₋₂, tᵢ₋₁) is the transition probability of a tag given the two preceding tags, which makes this a second-order (trigram) model. The product runs over all positions i, and t̂₁…t̂ₙ is the highest-scoring sequence. The Viterbi dynamic program finds it in time linear in sentence length and, for a trigram model, cubic in the number of tags.
TnT is a well-known example of this design. It uses tag trigrams for transitions and smooths them by interpolating unigram, bigram and trigram estimates, with weights set by deleted interpolation. It handles unseen words with suffix-based guessing (Brants 2000).
Discriminative taggers replace the generative product with a conditional model of each tag given a rich "history". That history includes nearby words, previous tags and spelling cues such as affixes, digits, capitals and hyphens. Such models are typically log-linear (maximum entropy) and decoded with beam search (Ratnaparkhi 1996).
A later refinement conditions each decision on tags to both the left and the right, using a cyclic dependency network. This network does not define a proper joint distribution, so the product of local scores is a score rather than a likelihood. That product can still be maximised exactly (Toutanova et al. 2003).
Across all these models, the practical difficulty is the same: unknown and rare words. Much of the engineering effort in taggers goes into handling them.
History and founding papers
Early statistical taggers built on annotated corpora. One was the Brown Corpus, compiled by Francis and Kučera in 1964 and only later given part-of-speech tags. Another was the Penn Treebank (Marcus et al., 1993). Church (1988) built an early stochastic tagger for unrestricted text, estimating its probabilities from the tagged Brown Corpus.
In 1992, two contrasting practical systems appeared:
- Cutting and colleagues built a first-order HMM trained with Baum-Welch on untagged text plus a lexicon. It grouped words into ambiguity classes, reducing a vocabulary of about 50,000 Brown-corpus words to roughly 400 classes. It exceeded 96% accuracy (Cutting et al. 1992).
- Brill showed that a rule-based tagger could rival stochastic ones. It started from each word's most frequent tag and then learned corrective patches. About 71 rules lowered the Brown Corpus error rate from about 7.9% to 5.1% (Brill 1992).
Brill then generalised this approach into transformation-based error-driven learning. It greedily learns an ordered list of readable rewrite rules. He proved such lists more expressive than decision trees built from the same queries (Brill 1995).
Machine-learning approaches diversified quickly:
- Ratnaparkhi's maximum entropy tagger scored 96.6% on held-out WSJ data (Ratnaparkhi 1996).
- Memory-based tagging stored labelled examples and classified new tokens by feature-weighted similarity. IGTree indexing made it fast (Daelemans et al. 1996).
- TnT showed that a carefully documented trigram HMM matches maximum entropy results and runs very fast (Brants 2000).
- Targeted linguistic features for capitalisation, verb forms and particles pushed a maximum entropy tagger to 96.86% (Toutanova & Manning 2000).
- Bidirectional conditioning with heavy lexicalisation and Gaussian-prior regularisation reached 97.24% per-token accuracy (Toutanova et al. 2003).
- SVM-based tagging reached 97.16% on WSJ (Giménez & Màrquez 2004).
| System (year) | Approach | Reported WSJ token accuracy |
|---|---|---|
| Ratnaparkhi (1996) | maximum entropy | 96.6% |
| Toutanova & Manning (2000) | enriched maximum entropy | 96.86% |
| Toutanova et al. (2003) | cyclic dependency network | 97.24% |
| Giménez & Màrquez (2004) | linear SVMs | 97.16% |
| Plank et al. (2016) | bi-LSTM, words + characters | 97.22% |
The data splits are not identical across these studies:
- The SVM tagger trained on WSJ sections 0–18 and tested on 22–24 (Giménez & Màrquez 2004).
- The enriched maximum entropy tagger trained on 0–20 and tested on 23–24 (Toutanova & Manning 2000).
- The 1996 tagger used its own contiguous partition (Ratnaparkhi 1996).
Gaps of a few hundredths of a point should therefore not be over-read.
For very large morphological tagsets, two techniques let HMM tagging beat TnT on German and Czech. Tags were factorised into attribute vectors, and contexts were estimated with probability estimation trees (Schmid & Laws 2008).
A parallel unsupervised line of work developed alongside the supervised taggers:
- Clark induced word classes from distributional and spelling information (Clark 2003).
- A fully Bayesian trigram HMM with Dirichlet priors beat maximum-likelihood training in every condition tested, by up to 14 accuracy points. Gains were largest under weak supervision. With a full tag dictionary, accuracy rose from 83.2% to 87.3% (Goldwater & Griffiths 2007).
- Projecting English tags through a bilingual graph reached 83.4% average accuracy on the twelve universal tags across eight languages, with no target-language annotation (Das & Petrov 2011).
How it is used today
Tagging is now usually one stage of a multilingual pipeline. UDPipe, for example, trains tokenisation, morphological analysis, tagging and parsing on CoNLL-U data (Straka et al. 2016). It was applied to Universal Dependencies 2.0 in a shared-task setting (Straka & Straková 2017).
For Czech, MorphoDiTa reached 95.75% on full tags of the Prague Dependency Treebank 2.5. It needed about 52MB of RAM, where the earlier Morče tagger needed 902MB. Its dictionary distinguishes 3,922 tags, an inventory separate from the Czech Academic Corpus set mentioned above (Straková et al. 2014).
Neural taggers have become standard. One example is a hierarchical bi-LSTM that combines word and character inputs. Its FREQBIN variant adds an auxiliary loss predicting each token's binned frequency. It reached a macro-averaged token accuracy of 96.52% over 22 Universal Dependencies v1.2 treebanks, and it helped most on out-of-vocabulary words (Plank et al. 2016).
Tagging is also often done jointly with other tasks:
- Dependency parsing: a transition-based system that assigns tags during SHIFT improved both tagging and parsing over a pipeline (Bohnet & Nivre 2012).
- Chinese word segmentation: character-based tagging clearly beat word-based tagging. Joint segmentation and tagging added only a marginal gain at much higher cost (Ng & Low 2004). Neural character models later removed most hand-built feature templates (Zheng et al. 2013).
- Arabic morphology: tokenisation, tagging and morphological disambiguation were combined by choosing among analyser outputs (Habash & Rambow 2005).
Non-standard text has its own line of tools. Twitter taggers use:
- tailored tagsets and features (Gimpel et al. 2011);
- Brown clusters learned from about 56 million tweets (Owoputi et al. 2013);
- vote-constrained bootstrapping of silver data (Derczynski et al. 2013).
Code-switched English–Spanish speech (Solorio & Liu 2008) and transliterated English–Hindi posts (Vyas et al. 2014) remain especially hard.
POS tags also serve as a diagnostic. Probing classifiers find that tag information is best recovered from the lower layers of neural translation encoders, whereas semantic tags benefit from deeper layers (Belinkov et al. 2018).
Open questions
The first question is whether English tagging is really solved. Token accuracy near 97% looks high, but only around 56% of WSJ sentences are tagged entirely correctly. Error analysis attributes a large share of remaining errors to inconsistent or mistaken gold annotation and unclear category definitions. For example, worth is tagged inconsistently as IN, JJ and NN across 114 treebank occurrences (Manning 2011). Ratnaparkhi had already observed an effect of annotator consistency. Training and testing on one annotator's text, rather than the mixed-annotator treebank, raised accuracy by about 0.5% (Ratnaparkhi 1996).
A related problem is consistency across treebanks of the same language that nominally follow shared guidelines. One response is a symmetric divergence measure over UPOS trigrams, which compares corpora without relying on any tagger (Aggarwal & Zeman 2020).
Several further issues remain open:
- Low-resource and noisy settings. A words-plus-characters bi-LSTM becomes competitive with the TnT HMM tagger at about 500 training sentences, or about 100 for non-Indo-European languages. It loses more accuracy than TnT once over 30% of labels are corrupted (Plank et al. 2016).
- Mixed-language input. In code-mixed text, errors in language identification and transliteration accumulate across pipeline stages (Vyas et al. 2014).
- Fine-grained morphology. Confusions such as nominative versus accusative case still dominate errors on German (Schmid & Laws 2008).
- Unsupervised evaluation. It is still unclear how best to measure induced classes against gold tags (Clark 2003; Goldwater & Griffiths 2007).
Key papers
Related concepts
- Universal Dependencies — Universal Dependencies is a framework and an open collection of treebanks that annotates the grammar of many languages with one shared…
- Dependency parsing — Dependency parsing is the task of recovering a sentence's syntactic structure as directed, labelled links between its words. Each word is…
- Named entity recognition — Named entity recognition locates stretches of text that refer to specific real-world things, such as people, organizations, places…
- Subword tokenization — Subword tokenization is a way of splitting text into units smaller than words but usually larger than single characters. A vocabulary of…
More papers
Topics: 🔤 Part-of-speech tagging · 🧩 Morphology
Shelves: 🗣️ Linguistics & languages · 💬 Natural language processing
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/part-of-speech-tagging/