📚 Corpus linguistics
Corpus linguistics is the empirical study of language through large, principled collections of naturally occurring text or transcribed speech (corpora). These are searched and counted with software to describe frequency, co-occurrence and variation, and to test claims about usage against real data.
What it is
A corpus is not just a pile of text. It is a collection chosen to represent some variety of language, stored in machine-readable form and often enriched with annotation. Corpus linguistics treats such collections as evidence. Questions about meaning, grammar, style or discourse are answered by observing what speakers and writers actually produce, rather than by relying only on introspection.
Corpora are commonly classified by type:
| Type | Purpose | Examples |
|---|---|---|
| General (reference) | Represent a language broadly across genres | Brown Corpus, British National Corpus (BNC), BCCWJ for Japanese (Asahara & Matsumoto 2016) |
| Specialised | Capture one domain or register | Stock-market news (Smadja 1993), terrorism reports from MUC-4 (Riloff & Shepherd 1997) |
| Parallel | Aligned texts in two or more languages | Canadian Hansards (Smadja et al. 1996), GALE Chinese–English news (Li et al. 2014) |
| Learner | Language produced by second-language learners, used to study interlanguage and errors | International Corpus of Learner English (ICLE), Cambridge Learner Corpus |
| Historical | Texts from earlier periods, used to trace language change | Helsinki Corpus, ARCHER |
The Brown Corpus is the classic general corpus of written American English, and it has served as a test bed. For example, a semantic concordance tagged every open-class word in 103 Brown passages with a WordNet sense (Miller et al. 1993; Miller et al. 1994). The BNC plays a similar reference role for British English. Its frequency list has been used as a baseline of "general English" against which domain vocabulary is measured (Mason 2004).
Balance across registers remains a design goal in newer resources. BCCWJ-DepPara annotates the core data of BCCWJ: roughly 1 to 1.2 million words in 1,980 samples from six registers, ranging from newspapers to blogs (Asahara & Matsumoto 2016). An Arabic readability corpus of 69,441 sentence-level segments, over a million words in total, was likewise assembled from many genres, topics and audiences (Elmadani et al. 2025).
The key idea
The central idea is that patterns of use become visible only in aggregate. Three tools carry most of the work.
A concordance lists every occurrence of a search term. In the key-word-in-context (KWIC) format, each hit is shown on one line with its left and right context aligned, so recurring neighbours stand out to the eye.
Frequency counts measure how often items occur. They are usually normalised, for example per million words, so that corpora of different sizes can be compared. Lexical diversity is often summarised with the type–token ratio:
TTR = V / N
Here V is the number of distinct word forms (types) and N is the total number of running words (tokens). The ratio falls as texts grow longer, so it is usually computed over fixed-size samples rather than over whole texts of unequal length.
Collocation measures test whether two words co-occur more often than chance would predict. The best-known association measure is the association ratio of Church and Hanks. It is an estimate of pointwise mutual information, computed for a single word pair rather than averaged over all pairs as in the information-theoretic quantity:
I(x, y) = log2 [ P(x, y) / ( P(x) · P(y) ) ]
P(x) and P(y) are the probabilities of words x and y in the corpus, estimated from their frequencies. P(x, y) is the probability that x is followed by y within a small window.
- A score near zero means the pair co-occurs about as often as independence predicts.
- A large positive score means the words attract each other.
Church and Hanks used a five-word window and discarded pairs seen five times or fewer. They found that scores above about 3 tended to be linguistically interesting. Unlike symmetric mutual information, they kept the ratio asymmetric so that word order is preserved. In 1988 Associated Press text of 44 million words, save…from occurred 176 times against 18 for the reverse order (Church & Hanks 1990).
Other association scores are also common:
- The Dice coefficient was used to align collocations across languages (Smadja et al. 1996).
- Log-likelihood was used to rank verb–argument fillers in the La Repubblica newspaper corpus of about 390 million Italian tokens (Lenci et al. 2010).
Contiguous word sequences, or n-grams, extend the same counting logic. Bigram statistics, for instance, were enough to rank candidate sentences in a text generator (Langkilde & Knight 1998).
Keyness applies frequency comparison to whole corpora. Words that are unusually frequent in a study corpus relative to a reference corpus are its "key" words. This is the logic behind selecting domain-characteristic verbs against BNC frequencies (Mason 2004).
History and founding papers
Corpus work has roots in pre-computer lexicography and concordance making. Computer-based corpus linguistics is usually dated from the Brown Corpus, compiled by Francis and Kučera in the 1960s. Firth's (1957) dictum that a word is known by the company it keeps supplied the theoretical motivation for collocation studies. Sinclair (1991) made concordance and collocation central to the study of lexis, laying the basis for what was later called the corpus-driven approach. The BNC was built in the 1990s as a large reference corpus for British English.
The statistical turn came around 1990. Church and Hanks showed that association norms, which psychologists had collected from human subjects, could be estimated automatically from newswire. They proposed these tables as an index for sorting concordance lines into senses (Church & Hanks 1990).
Smadja's Xtract added syntactic filtering to the statistics. On 10 million words of stock-market news, this filter raised precision from 40% to 80% at 94% recall (Smadja 1993). Its output then fed bilingual collocation translation from the Hansards. The test sets took 300 collocations from each of three Hansards years. Evaluators judged the translations correct 73% of the time on average, with results ranging from 65% to 78% (Smadja et al. 1996).
Yarowsky drew on a 380-million-word collection that included the Brown Corpus. He found that an ambiguous word keeps one sense per collocation in roughly 90–99% of cases (Yarowsky 1993).
In the same period, the semantic concordance showed how much sense annotation changes the picture of ambiguity. Of the open-class words listed in WordNet, 82.3% have a single sense. Yet only 27.2% of open-class tokens in running Brown text are monosemous (Miller et al. 1994).
How it is used today
Annotation is now central. Corpora carry part-of-speech tags, syntactic trees and dependencies (Asahara & Matsumoto 2016), coreference, and discourse relations. Because annotation embodies interpretive decisions, its reliability is measured with agreement statistics such as kappa.
- Untrained annotators labelled 1,412 definite descriptions in 33 Wall Street Journal articles. They reached K = 0.63 with detailed schemes and K = 0.76 with a two-way first-mention/subsequent-mention split (Poesio & Vieira 1998).
- In coherence annotation of 135 news texts, two annotators labelled each text independently, and agreement exceeded kappa 0.8. The resulting graphs showed crossed dependencies that tree-based discourse theories cannot represent (Wolf & Gibson 2005; Wolf & Gibson 2004).
Typical present-day uses include:
- Register and genre comparison. Spoken interviews and popular-science writing (22 texts, about 140,000 tokens) were found to differ significantly in pronoun use and in mention length (Amoia et al. 2012).
- Cross-lingual study through parallel data. Discourse labels were projected from English to Chinese using aligned news (Li et al. 2014).
- Social-media and pragmatics corpora. One example is irony annotation of French, English and Italian tweets; the English part alone holds 5,173 ironic and 6,116 non-ironic tweets (Karoui et al. 2017). Another is a set of 382,210 election tweets, of which 3,000 were hand-labelled for stance and offensiveness (Grimminger & Klinger 2021).
- Lexicon building from co-occurrence. Applications range from bootstrapped semantic categories (Riloff & Shepherd 1997) to general distributional resources stored as word–link–word tensors (Baroni & Lenci 2010).
- Fixed reference datasets for reproducible research. One example is a frozen snapshot of 10,921 computational-linguistics papers taken in February 2007 (Bird et al. 2008).
- Balanced test sets that expose bias. A pronoun corpus of 8,908 pairs, balanced by gender, revealed weaker performance on feminine examples (Webster et al. 2018).
On the software side, widely used tools include concordancers such as WordSmith Tools and AntConc. Web platforms such as Sketch Engine and CQPweb add corpus query languages over annotated corpora: CQL in Sketch Engine and CQP syntax in CQPweb. Programming libraries such as NLTK support custom analyses.
Open questions
Representativeness is still hard to define. No sample can mirror "a language" as a whole, and balance decisions shape every frequency reported.
Each association measure has its own biases. Mutual information becomes unreliable for rare pairs, which is why low-count pairs are filtered out. Purely distributional scores also cannot separate compositional combinations from idioms (Church & Hanks 1990).
Annotation raises its own problems. Human labels can diverge even on antecedents, which casts doubt on evaluation against a single gold standard (Poesio & Vieira 1998). In discourse annotation, forcing coherence relations into trees would hide the crossed dependencies and multi-parent nodes that graph-based annotation reveals (Wolf & Gibson 2005).
Web-scale text blurs the line between curated corpora and opportunistic crawls. Models trained on loosely filtered web data reproduce toxic and biased language (Schick et al. 2021), and skewed benchmarks under-represent some groups (Webster et al. 2018).
Finally, many phenomena depend on context the text does not contain. Implicit irony is one example (Karoui et al. 2017), and offensive speech that requires world knowledge is another (Grimminger & Klinger 2021). Such cases mark the limits of what counting alone can reveal.
Key papers
Related concepts
- Zipf's law — Zipf's law is the empirical regularity that, when the items of a collection such as the words of a text are ranked by frequency, each…
- Distributional semantics — Distributional semantics is the study and modelling of meaning through the statistics of linguistic context. Words, phrases or sentences…
- Lexicography — Lexicography is the discipline of compiling, describing and designing dictionaries and related lexical resources. It covers both the…
- Universal Dependencies — Universal Dependencies is a framework and an open collection of treebanks that annotates the grammar of many languages with one shared…
More papers
Topics: 🗂️ Treebanks · ⏳ Historical linguistics & language change
Shelves: 🗣️ Linguistics & languages
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/corpus-linguistics/