🌫️ Distributional semantics
Distributional semantics is the study and modelling of meaning through the statistics of linguistic context. Words, phrases or sentences are represented by the contexts they occur in, on the assumption that expressions sharing comparable contexts are likely to be close in meaning.
What it is
Distributional semantics treats meaning as something that can be read off usage. A word is described by where it appears: which neighbouring words, syntactic partners or documents surround it. Each word is then encoded as a vector of context counts, or of weights derived from them. Closeness between vectors stands in for semantic relatedness.
The field covers four main concerns:
- how to define and weight contexts,
- how to measure similarity between context profiles,
- how to compress sparse count tables,
- how to extend word-level representations to phrases and sentences.
It sits between corpus linguistics, computational linguistics, cognitive science, lexicography and information retrieval. Its methods underpin tasks such as word sense disambiguation (Basile et al. 2014) and the detection of semantic change (Gulordava & Baroni 2011).
The key idea
The central mechanism has three steps. Co-occurrence counts are turned into association weights, each word becomes a vector of those weights, and pairs of vectors are compared geometrically. A common weighting is pointwise mutual information, popularised for corpora as the association ratio (Church & Hanks 1990). Cosine similarity is the most common comparison:
PMI(w, c) = log2 [ P(w, c) / ( P(w) · P(c) ) ]
cos(u, v) = Σᵢ uᵢ·vᵢ / ( √(Σᵢ uᵢ²) · √(Σᵢ vᵢ²) )
In the first line, w is a target word and c a context. The context may be another word inside a window, a syntactic relation plus a word, or a document. P(w, c) is the probability of observing them together, and P(w) and P(c) are their separate probabilities. PMI is high when the pair co-occurs more often than chance would predict. It is negative when the pair co-occurs less often than chance, and undefined for pairs never seen together. For this reason positive PMI (PPMI), which sets negative values to zero, is the usual variant.
In the second line, u and v are the context vectors of two words, uᵢ and vᵢ are their weights on context dimension i, and the sum runs over all dimensions. Cosine ranges from −1 to 1 in general, and from 0 to 1 when all weights are non-negative, as with counts or PPMI. A value of 1 means the vectors point in the same direction.
Church and Hanks counted y occurring after x within a window of 5 words. They discarded pairs seen 5 times or fewer and treated scores above roughly 3 as interesting (Church & Hanks 1990). Later work often weights counts with local mutual information, which multiplies association by frequency (Bruni et al. 2012; Gulordava & Baroni 2011).
Many variants refine these steps:
- Count matrices can be reduced with truncated SVD, as in Latent Semantic Analysis.
- They can also be factorised probabilistically. PLSA explains each word–document co-occurrence through a hidden topic z and fits the model by EM (Hofmann 1999).
- Cosine is not the only option. Lee compared seven measures and proposed the asymmetric skew divergence. With α = 0.99 it gave about 4% average error reduction against Jensen-Shannon divergence, across five test partitions of unseen pairs (Lee 1999).
- Weeds and Weir reframed similarity as retrieval of one word's features by another, scored with precision and recall. Several known measures fall out as special parameter settings of this framework (Weeds & Weir 2003; Weeds & Weir 2005).
History and founding papers
The distributional hypothesis is usually traced to Zellig Harris (1954) and to J. R. Firth (1957). Latent Semantic Analysis, associated with Deerwester and colleagues (1990) and Landauer and Dumais (1997), made dimensionality-reduced co-occurrence spaces practical.
On the computational side, Church and Hanks showed in 1990 that mutual-information statistics over newswire could estimate word-association norms automatically, instead of eliciting them from people at great cost. Their corpora included 1987 AP text of 15M words and 1988 AP text of 36M words. The top associates of doctor resembled classic psycholinguistic norms (Church & Hanks 1990).
The 1990s brought a probabilistic turn. Hofmann's PLSA replaced LSA's squared-error fit with likelihood and produced factors that separate senses of ambiguous words such as "segment" or "matrix" (Hofmann 1999). In the same year, Lee used distributionally similar words to estimate probabilities of unseen verb–object pairs. She tested them on a binary pseudo-disambiguation-style choice over such pairs. All measures she compared stayed well below the 51% error rate of Katz backoff on that task (Lee 1999).
The 2000s and early 2010s focused on the nature of similarity itself. Weeds and Weir found that thesaurus generation favours high recall while pseudo-disambiguation favours high precision. They traced this to an asymmetry in lexical substitutability (Weeds & Weir 2005). Erk and Padó gave words relation-specific selectional-preference vectors so that meaning could shift with syntactic context (Erk & Padó 2008). Reisinger and Mooney replaced the single centroid per word with several prototypes obtained by clustering occurrences (Reisinger & Mooney 2010). Turney and Pantel's 2010 JAIR article became a widely cited account of vector space models of semantics (Turney & Pantel 2010).
Around 2010 composition became a central theme:
- Guevara learned adjective–noun composition by partial least squares regression onto observed phrase vectors (Guevara 2010).
- Zanzotto and colleagues estimated full matrices A and B for the additive form z = Ax + By. They used least-squares regression on dictionary-derived examples (Zanzotto et al. 2010).
- Clarke proposed an algebraic framework in which many composition methods appear as special cases and each acquires an entailment ordering (Clarke 2012).
- Baroni and co-authors showed that corpus-derived phrase vectors carry entailment signals. A quantifier classifier generalised to unseen quantifiers (Baroni et al. 2012).
Evaluation matured alongside these models. BLESS profiled models by typed relations rather than a single score (Baroni & Lenci 2011). SICK provided about 10,000 sentence pairs for testing composition (Marelli et al. 2014; Marelli et al. 2014).
Prediction-based neural embeddings, popularised by Mikolov and colleagues (2013), then reshaped the field. Levy, Goldberg and Dagan carried lessons from word embeddings back into count-based distributional similarity (Levy et al. 2015).
How it is used today
Distributional representations appear across language technology and computational linguistics:
- Word sense disambiguation. Replacing literal gloss overlap with cosine in an LSA space let an enhanced Lesk method beat all SemEval-2013 participants for English (Basile et al. 2014).
- Idiom and multiword detection. A low cosine between a composed vector and the observed vector of an expression signals non-compositionality (Katz & Giesbrecht 2006).
- Semantic change. Each word's vectors from the 1960s and the 1990s were compared in Google Books 2-grams. The resulting scores correlated with the change ranking of five human raters over 100 words at 0.386, against 0.301 for a frequency signal (Gulordava & Baroni 2011).
- Topic model quality. Cosine in a topic-word space reached Spearman correlations of 0.76–0.80 with human coherence ratings. The topics came from the NYT, 20 Newsgroups and Genomics collections (Aletras & Stevenson 2013).
- Multimodal semantics. Combining text vectors with visual vectors helps, especially for colour terms (Bruni et al. 2012).
- Retrieval-style tasks. Tf-idf vector spaces have been used to disambiguate same-name entities across documents (Bagga & Baldwin 1998). LSA spaces enriched with pseudo-word features have been used to classify dialogue acts (Serafin & Eugenio 2004).
- Relations and knowledge bases. Relation encoders trained on contexts with the entity mentions blanked out bring a distributional notion of similarity to relation learning (Soares et al. 2019). Compositional vector space models have been applied to knowledge base completion (Neelakantan et al. 2015).
The hypothesis also informs how modern pretrained models are interpreted. RoBERTa pretrained on word-shuffled text and then fine-tuned on normal text lost surprisingly little downstream accuracy. The authors read this as evidence that much of masked language modelling's benefit comes from co-occurrence statistics rather than word order (Sinha et al. 2021).
Open questions
Several problems remain unresolved.
Relation type. Cosine conflates very different relations, so dog lies close to both cat and animal. Directional measures based on distributional inclusion help separate hypernyms from coordinates (Lenci & Benotto 2012).
The table below shows mean average precision on a BLESS subset. The subset covers 200 concrete nouns from 17 classes, each paired with coordinates, hypernyms, meronyms and random nouns. Each column gives how well a measure ranks relata of that relation near the top. A good hypernym detector should score high on HYPER relative to COORD.
| Measure | COORD | HYPER |
|---|---|---|
| Cosine | 0.79 | 0.23 |
| invCL | 0.38 | 0.40 |
Even invCL ranks hypernyms only narrowly above coordinates (Lenci & Benotto 2012). Model behaviour also depends on the type of context. Narrow windows favour coordinates, while document contexts separate related from unrelated words but distinguish relation types poorly (Baroni & Lenci 2011).
Composition. There is still no consensus on how word vectors should combine. In Guevara's experiments, learned composition came closest to observed phrase vectors, yet simple addition did best on neighbour overlap (Guevara 2010). In SemEval-2014, the best systems reached about 0.82 Pearson r on relatedness and 84.5% accuracy on entailment. Purely compositional systems exceeded 0.70 r and 70% accuracy, around the field average, and the leaders also used non-compositional cues (Marelli et al. 2014).
Idiomaticity. Static and contextualised models largely fail to register that opaque noun compounds resist substitution of their parts (Garcia et al. 2021).
Polysemy. Occurrence clusters capture ambiguity, but they are finer than dictionary senses and sometimes odd (Reisinger & Mooney 2010).
Benchmarks. In SICK, a participating team found that negation words alone identified 86.4% of contradiction pairs, which shows that SICK can reward shortcuts (Marelli et al. 2014). Probes of pretrained models may likewise overstate their syntactic knowledge (Sinha et al. 2021).
Theory. Two broader questions stay open: how far distributional meaning can support logical inference (Clarke 2012; Baroni et al. 2012), and how much grounding beyond text is needed (Bruni et al. 2012).
Key papers
Related concepts
- Word embeddings — Word embeddings are dense, real-valued vectors that represent the words of a vocabulary as points in a shared geometric space. They are…
- Semantic similarity — Semantic similarity is a graded measure of how close two linguistic units, such as words, concepts, sentences or documents, are in meaning.…
- Lexical semantics — Lexical semantics is the study of word meaning. It asks what words and multiword expressions denote, how a word's senses are organized, how…
- Corpus linguistics — Corpus linguistics is the empirical study of language through large, principled collections of naturally occurring text or transcribed…
More papers
Topics: 📖 Lexical semantics & word embeddings
Shelves: 🗣️ Linguistics & languages · 💬 Natural language processing
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/distributional-semantics/