🧤 GloVe
GloVe is an unsupervised method that learns a fixed vector for every word by fitting dot products of word and context vectors to the logarithms of global word co-occurrence counts. Words used in similar contexts get nearby vectors, and many relations between words become consistent vector offsets.
What it is
Global Vectors for Word Representation (GloVe) is a way to turn each word of a vocabulary into a list of numbers, typically between 50 and 300 of them. Words that appear in similar surroundings end up with similar lists. It belongs to the family of word embeddings and rests on the old idea of distributional semantics: a word's meaning can be read off the company it keeps. Jeffrey Pennington, Richard Socher and Christopher Manning introduced it in 2014. They released the training code and ready-made vectors, and these quickly became a default input for natural language processing (NLP) systems (Pennington et al. 2014).
GloVe gives a word the same vector wherever it occurs. That makes it a static embedding, unlike the contextual representations that came later. A GloVe vector is cheap to look up, simple to compare by cosine similarity, and often good enough to tell that "frog" is near "toad" or that "Paris" relates to "France" as "Rome" relates to "Italy".
The key idea
Ratios, not raw probabilities
The authors start from a table X that records, for every pair of words, how often one appears near the other in a corpus. Their central observation is that a single co-occurrence probability says little about meaning, while the ratio of two probabilities says a lot (Pennington et al. 2014). Take "ice" and "steam" and compare how likely various probe words are to appear near each. For "solid", the ratio P(solid | ice) / P(solid | steam) is about 8.9. For "gas" it is about 0.085. For probe words that relate equally to both, such as "water", or to neither, such as "fashion", the ratio stays near 1 (Pennington et al. 2014). The ratio therefore singles out the dimension of meaning that separates the two words and cancels the noise of words that are unrelated to the contrast.
The objective
To make vector differences encode these ratios, GloVe asks the dot product of a word vector and a context vector to match the log co-occurrence count. The fit is done by weighted least squares:
J = Σ_{i,j} f(X_ij) · (w_iᵀ w̃_j + b_i + b̃_j − log X_ij)²
f(x) = (x / x_max)^α if x < x_max
f(x) = 1 otherwise
Here X_ij is the number of times word j appears in the context window of word i. The vector w_i belongs to word i in its role as a target, and w̃_j to word j in its role as a context. The scalars b_i and b̃_j are per-word biases that absorb how frequent each word is. The sum runs only over pairs with X_ij > 0, so the logarithm is always defined and the many empty cells cost nothing (Pennington et al. 2014).
The weighting function f does two jobs. It reduces the weight of rare pairs, whose counts are noisy, and it caps the weight of very frequent pairs, so that common function words do not dominate the fit. The threshold x_max and the exponent α are hyperparameters. In the original paper α is a fraction below one, so f grows more slowly than the count itself (Pennington et al. 2014). After training, the word and context vectors can be summed into one embedding, w + c. This variant is recommended for GloVe and for skip-gram with negative sampling alike (Levy et al. 2015).
Why linear analogies appear
A theoretical account by Arora and colleagues models text as generated by a slowly drifting discourse vector. It assumes the word vectors are isotropic, meaning they are spread evenly in all directions with no preferred one. Under that assumption it derives closed-form links between co-occurrence statistics and vector inner products. These links justify PMI-, word2vec- and GloVe-style objectives and some of GloVe's hyperparameter choices (Arora et al. 2016). In the same work, the difference vectors within one analogy relation, such as country–capital, share a single main direction. Their average cosine with the first singular vector is 0.51, against 0.035 with the second, and GloVe and word2vec spaces behave the same way (Arora et al. 2016).
History and founding papers
Two traditions it set out to merge
Before GloVe, word vectors came from two lines of work. Count-based methods factorize a global co-occurrence matrix, as in latent semantic analysis (LSA). LSA was introduced by Deerwester and colleagues in 1990 and is also called latent semantic indexing. Such methods use corpus statistics efficiently but do poorly on analogies (Pennington et al. 2014).
Prediction-based methods learn from local windows. Mikolov and co-authors proposed CBOW, which reconstructs a word from its neighbours, and skip-gram, which predicts neighbours from a word. They also built an analogy test solved by vector arithmetic (Mikolov et al. 2013). A follow-up added negative sampling and subsampling of frequent words, and the code was released as word2vec (Mikolov et al. 2013). These models handled analogies well but never looked at global counts directly. GloVe was designed to get the strengths of both (Pennington et al. 2014).
Reported results
The headline results in the founding paper were these (Pennington et al. 2014):
- 75% accuracy on the word analogy task.
- Better scores than competing models on word similarity tests and on named entity recognition (NER).
- In a controlled comparison with word2vec, a better result reached in less training time. Both systems used a 6-billion-token corpus of Wikipedia 2014 plus Gigaword 5, a 400,000-word vocabulary, 300-dimensional vectors and a symmetric window of 10.
- With 50-dimensional vectors on NER, F1 of 93.2 on the development set, 88.3 on test, 82.9 on ACE and 82.2 on MUC7. These scores beat CBOW, Hellinger PCA (HPCA), Collobert–Weston (CW) and SVD baselines.
The paper also found that accuracy rises only slightly beyond roughly 200 dimensions. Small asymmetric windows suit syntactic analogies, and wider windows suit semantic ones (Pennington et al. 2014).
Was it the algorithm or the tuning?
Levy, Goldberg and Dagan reopened the count-versus-predict debate. Under identical tuning, PPMI, SVD, skip-gram with negative sampling and GloVe differ only locally or insignificantly. A single hyperparameter often matters more than the choice of algorithm (Levy et al. 2015). Schnabel and colleagues trained GloVe and five other models under matched conditions. The models ranked differently depending on the criterion, so there is no single "best" embedding (Schnabel et al. 2015). They also found that all six embeddings carry word-frequency information, which distorts cosine semantic similarity. For GloVe, a simple classifier told frequent from rare words with close to 100% accuracy at frequency cut-offs up to 1000 (Schnabel et al. 2015).
How it is used today
Input features and baselines
For years GloVe served as a common input layer of neural NLP models, where each word simply received its pretrained vector (McCann et al. 2017). Such pretrained tables are often fine-tuned during training. In opinion target extraction, fine-tuning SENNA vectors in an Elman network added 13.01 points absolute on the Laptop data (Liu et al. 2015). Simple models that only average or max-pool pretrained vectors match or beat LSTM and CNN encoders on many classification and matching tasks (Shen et al. 2018).
Generic co-occurrence vectors can place "good" next to "bad". The same weakness had motivated sentiment-aware embeddings, both before GloVe and in the same year (Maas et al. 2011; Tang et al. 2014). Other variants add word order, which plain skip-gram ignores (Song et al. 2018). Subword models such as fastText follow an idea close to subword tokenization. They build vectors from character n-grams and so cover rare and unseen words that GloVe cannot represent (Bojanowski et al. 2017). fastText vectors were later released for 157 languages (Grave et al. 2018).
Cross-lingual and historical work
Static spaces from separate languages can be aligned by an orthogonal map (Xing et al. 2015; Artetxe et al. 2016; Smith et al. 2017). The supervision can shrink to almost nothing (Artetxe et al. 2017) or vanish entirely (Artetxe et al. 2018). The same idea extends to dozens of languages in one space (Ammar et al. 2016); see multilingual NLP. For low-resource machine translation, pretrained source embeddings help most when the baseline is weak but still trainable (Qi et al. 2018). Aligned embeddings from successive decades can also measure language change. The best-known study built them with PPMI, SVD and skip-gram rather than GloVe (Hamilton et al. 2016).
After contextual embeddings
CoVe fed GloVe vectors into a translation encoder and appended the encoder's contextual outputs to them. The combination outperformed GloVe alone (McCann et al. 2017). Bidirectional language models (biLMs) built on LSTMs, the Transformer or gated CNNs all clearly beat GloVe when fed into the same task models (Peters et al. 2018):
| Task (test metric) | GloVe | 2-layer LSTM biLM | Transformer biLM | Gated CNN biLM |
|---|---|---|---|---|
| MultiNLI matched (accuracy) | 77.0 | 79.6 | 79.4 | 78.3 |
| SRL (F1) | 81.4 | 84.6 | 84.1 | 84.1 |
| Constituency parsing (F1) | 91.8 | 93.9 | 93.7 | 92.9 |
| NER (F1) | 89.9 | 91.7 | 91.1 | 91.2 |
A linear classifier that labels constituency spans from the input features alone reaches only 18.1% F1 with GloVe, against nearly 80% with biLM layer activations (Peters et al. 2018). Today GloVe survives mainly as a fast baseline, a teaching example and a probe target, rather than as the core of state-of-the-art systems.
Open questions
Bias is the most studied problem. Word embeddings absorb stereotypes, such as analogies pairing women with "homemaker" (Bolukbasi et al. 2016). GN-GloVe changes the GloVe objective to confine gender to one reserved dimension (Zhao et al. 2018). Yet clustering and classifier tests show that much gendered structure survives this and other debiasing methods (Gonen & Goldberg 2019). In coreference resolution, contextual ELMo vectors show stereotype gaps at least as large as GloVe's (Zhao et al. 2019); see bias, fairness and safety.
Other questions remain open:
- How to evaluate static vectors fairly when intrinsic and extrinsic rankings disagree (Schnabel et al. 2015).
- How much the count-based framing matters once hyperparameters are tuned (Levy et al. 2015).
- How far the theory of linear analogies extends beyond its isotropy assumption (Arora et al. 2016).
Static vectors assign each word a single vector regardless of context. They therefore cannot separate the senses of an ambiguous word, a limit that matters for word sense disambiguation (McCann et al. 2017). More on word meaning and vectors is collected under lexical semantics.
Key papers
Related concepts
- Word embeddings — Word embeddings are dense, real-valued vectors that represent the words of a vocabulary as points in a shared geometric space. They are…
- Distributional semantics — Distributional semantics is the study and modelling of meaning through the statistics of linguistic context. Words, phrases or sentences…
- Latent semantic indexing — Latent semantic indexing (LSI) is a retrieval and text-analysis method that factorizes a weighted term–document matrix with a truncated…
- Semantic similarity — Semantic similarity is a graded measure of how close two linguistic units, such as words, concepts, sentences or documents, are in meaning.…
More papers
Topics: 📖 Lexical semantics & word embeddings
Shelves: 🤖 Artificial Intelligence
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/glove/