🖊️ Linguistic annotation
Linguistic annotation is the practice of adding explicit, interpretive labels to language data, such as word classes, syntactic structure, meanings, reference or attitude, so that a corpus can be searched, studied and used to train and test language technology.
What it is
A raw text is a sequence of characters. Linguistic annotation adds a layer of explicit judgement on top of it. One label says "this is a noun." Another says "these two words refer to the same person." A third says "this sentence expresses an opinion." The result is an annotated corpus, the basic resource of corpus linguistics and of supervised natural language processing (NLP). The labels are interpretations, not observations. Two careful people can disagree, and much of the craft of annotation is about controlling that disagreement.
Annotation is usually organised in layers. Each layer captures one kind of linguistic fact and builds on the layers below it:
| Layer | "Mary sold her old car to John yesterday." |
|---|---|
| Tokenization | Mary · sold · her · old · car · to · John · yesterday · . |
| Part-of-speech tagging | NNP VBD PRP$ JJ NN TO NNP NN . |
| Syntax (dependencies) | sold is the root; Mary is its subject; car is its object, with modifiers her and old |
| Semantic role labeling | Arg0 = Mary, Arg1 = her old car, Arg2 = to John, ArgM-TMP = yesterday |
| Coreference resolution | her → Mary |
| Discourse analysis | a possible contrast or cause relation with a neighbouring sentence |
| Sentiment | neutral, factual report |
The part-of-speech tags in the example follow the Penn Treebank tagset, and the role labels follow PropBank (Marcus et al. 1993; Palmer et al. 2005). Other layers include named entity recognition, word sense disambiguation, entity links to Wikipedia (Ferragina & Scaiella 2010) and grammatical error categories in learner writing (Dahlmeier et al. 2013; Bryant et al. 2017).
The key idea
Annotation as a model of language
An annotation scheme is a theory in miniature. It decides which distinctions count and how they relate to one another. The Penn Treebank designers deliberately shrank the Brown Corpus tagset. They left out any distinction recoverable from the word itself or from its place in the tree. This reduced both sparse data and inconsistencies between annotators (Marcus et al. 1993). For German, whose word order is freer, Skut and colleagues abandoned English-style phrase structure with traces. Their scheme used flat trees with crossing branches, with grammatical functions stored on the edges (Skut et al. 1997).
The MATTER cycle
Pustejovsky and colleagues describe annotation projects as a loop called MATTER (Pustejovsky & Moszkowicz 2012):
- Model the phenomenon.
- Annotate texts with that model.
- Train a system on the labels.
- Test it.
- Evaluate the results.
- Revise the model and the guidelines.
Testing the model is not an afterthought. In the development of the ISO-Space standard for spatial language, the authors treated model testing as part of how the standard itself took shape (Pustejovsky & Moszkowicz 2012). In practice the loop starts with written guidelines and pilot rounds on small samples. The guidelines are then revised wherever annotators diverge. Wiebe and colleagues used statistical models of judges' bias to guide such a revision of a subjectivity manual, and pairwise agreement rose by up to 16 kappa points (Wiebe et al. 1999).
Measuring agreement
Raw percent agreement is misleading because some agreement happens by chance. With a 95%/5% split between two categories, chance alone yields roughly 90.5% agreement (Artstein & Poesio 2008). Cohen's kappa corrects for this:
κ = (Pₒ − Pₑ) / (1 − Pₑ)
Here Pₒ is the proportion of items on which two coders agree, and Pₑ is the agreement expected by chance. In Cohen's version, Pₑ is computed from each coder's own label distribution. A κ of 1 means perfect agreement, and 0 means agreement no better than chance. Krippendorff's alpha generalises the idea to any number of coders, to missing data and to graded distances between labels:
α = 1 − Dₒ / Dₑ
Dₒ is the observed disagreement and Dₑ is the disagreement expected by chance. Both are weighted by how different the clashing labels are. Artstein and Poesio clarified that the "kappa" common in computational linguistics is really a multi-coder extension of Scott's π, not Cohen's κ. They recommended α for set-valued tasks such as anaphora, and suggested 0.8 as a cautious threshold rather than a fixed cutoff (Artstein & Poesio 2008).
History and founding papers
Treebanks and propbanks
The modern era of large-scale annotation is usually dated to the Penn Treebank. It holds over 4.5 million words of American English, all tagged for part of speech, and more than half of it carries skeletal syntactic trees (Marcus et al. 1993). Its tagset has 36 word tags plus 12 for punctuation and currency, against 87 simple tags in the Brown Corpus. The project also showed that having a program pre-annotate and humans correct is faster and more consistent than purely manual work (Marcus et al. 1993). The treebank went on to support work such as Petrov and colleagues' latent refinement of its categories, an automatic "annotation" of tree symbols that reached 90.2% F1 (Petrov et al. 2006).
PropBank added predicate–argument structure on top of the Wall Street Journal trees. It defined roles verb by verb in frames files and became the standard training data for role labelling (Palmer et al. 2005). OntoNotes then pursued a multi-layer resource under the banner of a "90% solution" (Hovy et al. 2006). Even so, it reported only percent agreement (Artstein & Poesio 2008).
Toward shared schemes
Divergent treebank conventions made cross-lingual comparison unreliable. McDonald and colleagues, with Joakim Nivre among them, harmonised six languages under one dependency scheme. This was the direct forerunner of Universal Dependencies (McDonald et al. 2013). A carefully checked English dependency gold standard followed (Silveira et al. 2014).
The crowd arrives
Snow, O'Connor, Jurafsky and Ng tested Amazon Mechanical Turk workers on five tasks that already had expert labels. On average, about four non-expert labels per item matched one expert, and 7,000 affect labels cost US$2.00 (Snow et al. 2008). Quality control quickly became a research topic of its own. For image captions, requiring US residence and a qualification test raised unanimous expert approval to 82.0%, against 60.2% for unscreened writers (Rashtchian et al. 2010).
Crowdsourcing soon produced very large datasets: natural language inference pairs (Bowman et al. 2015), science questions through guided writing (Welbl et al. 2017), and dense scene graphs mapped to WordNet (Krishna et al. 2016).
Aggregating noisy labels
Majority voting is easily fooled by careless workers. MACE (Multi-Annotator Competence Estimation) treats the true label as hidden and gives each annotator a trustworthiness parameter. It is trained with EM or Variational Bayes (Hovy et al. 2013). Its trust estimates correlated with true annotator skill at about 0.90, while Cohen's κ fell as low as 0.13 on one task (Hovy et al. 2013). Earlier, Wiebe and colleagues had derived bias-corrected tags from a latent class model, an ancestor of the same idea (Wiebe et al. 1999).
How it is used today
Tools and formats
Annotation is done in dedicated software. brat is a browser tool for spans, relations and events. In one experiment, its light machine assistance cut total annotation time by about 15% (Stenetorp et al. 2012). INCEpTION adds recommenders that retrain as the annotator works, optional active learning and links to knowledge bases. It exchanges data with external components as UIMA CAS XMI (Klie et al. 2018). Skut's German tool already used a statistical tagger to propose function labels, which cut time per sentence by roughly a quarter (Skut et al. 1997). Column formats in the CoNLL tradition remain the common currency for dependency data (McDonald et al. 2013).
New domains and modalities
Each new genre needs its own scheme. A Twitter tagset added categories for hashtags, mentions and emoticons, and reached κ = 0.914 on doubly annotated tweets (Gimpel et al. 2011). Multimodal sentiment data now labels text, audio and video separately, because a single shared label can hide differences between modalities (Yu et al. 2020). Evaluation also relies on annotation. Translation edit rate, for example, can be computed against human-targeted references (Snover et al. 2006).
Annotation by language models
Large language models are now used as annotators. With unchanged codebooks, zero-shot ChatGPT beat filtered crowd workers by about 25 percentage points of accuracy on tweet and news coding, at roughly a thirtieth of the cost (Gilardi et al. 2023). Crowdsourcing instructions themselves have become training material for models that generalise across tasks (Mishra et al. 2022). In mathematical reasoning, step-level labels can be generated automatically by checking where model continuations lead. A reward model trained on such labels outperformed one trained on human step labels (Wang et al. 2024).
Open questions
Is disagreement noise or signal? On hate speech, crowd majority labels agreed weakly with activist experts (κ = 0.34), and models trained on expert labels scored higher (Waseem 2016). Davani and colleagues argue that majority voting erases legitimate minority views. They model each annotator separately instead, and the spread between those models' predictions tracks real disagreement (Davani et al. 2022). This challenges the single gold standard on which agreement statistics rest (Artstein & Poesio 2008).
A second problem is artefacts. The way annotators are prompted can leave regularities in the data that models exploit without understanding the task. This has been shown for inference corpora (Gururangan et al. 2018) built through crowdsourcing (Bowman et al. 2015).
Third, machine annotation raises questions of validity. Model labels can be highly consistent from run to run and still go wrong on new material. Their quality also depends on prompt wording (Gilardi et al. 2023). Automatically generated labels also carry noise of their own (Wang et al. 2024).
Finally, the classic tensions remain. Expert depth competes with crowd scale. Universal schemes compete with language-specific detail (McDonald et al. 2013). And there is still no settled answer to how much agreement is "enough" for a given use.
Key papers
Related concepts
- Universal Dependencies — Universal Dependencies is a framework and an open collection of treebanks that annotates the grammar of many languages with one shared…
- Corpus linguistics — Corpus linguistics is the empirical study of language through large, principled collections of naturally occurring text or transcribed…
- Semantic role labeling — Semantic role labeling is the task of finding, for each predicate in a sentence, the phrases that act as its participants and labeling each…
More papers
Topics: 🗂️ Treebanks · 📏 Evaluation & benchmarks
Shelves: 🗣️ Language and Linguistics · 🤖 Artificial Intelligence
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/annotation/