🧮 Word embeddings
Word embeddings are dense, real-valued vectors that represent the words of a vocabulary as points in a shared geometric space. They are usually learned from co-occurrence patterns in large unlabeled corpora, so that words used in similar contexts end up close to one another.
What it is
Classical NLP systems treat each word as a one-hot symbol, an index with no relation to any other index. Under this encoding, rare words receive poorly estimated parameters and unseen words receive none (Turian et al. 2010). A word embedding replaces the symbol with a vector of a few dozen to a few hundred dimensions. Similarity between words then becomes a geometric quantity, typically cosine similarity.
Typical sizes vary widely. Huang et al.'s global-context model uses 50 dimensions over a 30,000-word vocabulary (Huang et al. 2012). The public word2vec model trained on Google News holds roughly 3 million English words and phrases, each a 300-dimensional vector (Bolukbasi et al. 2016).
The idea rests on the distributional hypothesis, associated with Harris (1954) and Firth (1957): a word's meaning is reflected in the company it keeps. Embeddings turn that intuition into a trainable object. They can be learned once from raw text and then plugged into almost any supervised system as extra input features (Turian et al. 2010).
The key idea
A widely used training mechanism is the skip-gram model with negative sampling. In this model, each word must predict the words that appear near it (Mikolov et al. 2013; Mikolov et al. 2013). The learned space is commonly evaluated with vector arithmetic on analogies (Mikolov et al. 2013).
J(w, c) = log σ(v′_c · v_w) + Σ_{i=1..k} log σ(−v′_{n_i} · v_w)
answer(a : b :: c : ?) = argmax_{x ∉ {a, b, c}} cos(v_x, v_b − v_a + v_c)
In the first line, w is a target word and c is a word observed inside its context window. v_w is the "input" vector of w, and v′_c is the "output" vector of the context word c. The function σ is the logistic sigmoid, and the dot product measures how compatible the two vectors are. The sum runs over k "negative" words n_i, drawn at random from a noise distribution. Training raises J for each observed (w, c) pair. This pulls true context pairs together and pushes random pairs apart.
In the second line, a, b and c are the given words of an analogy, and v denotes their vectors. The argmax runs over vocabulary words x other than a, b and c. These three are excluded because one of them would otherwise usually win. cos is cosine similarity. The answer is the word whose vector lies nearest to the offset vector v_b − v_a + v_c.
The striking empirical finding was that linguistic relations behave roughly like constant directions in the space (Mikolov et al. 2013).
Count-based methods reach similar spaces from a different starting point. GloVe fits word vectors directly to global co-occurrence statistics (Pennington et al. 2014). Sparse PPMI vectors built from the same corpus and window as skip-gram recover a comparable amount of analogical structure. The parity is clearest when analogies are scored with the multiplicative rule described next (Levy & Goldberg 2014). The same analysis showed that the additive analogy rule is algebraically a combination of three ordinary word similarities. On that basis it proposed a multiplicative variant, 3COSMUL (Levy & Goldberg 2014). Levy and Goldberg concluded that neural embeddings mainly preserve regularities already present in co-occurrence counts. Under the additive rule, however, the explicit vectors trail somewhat (Levy & Goldberg 2014).
History and founding papers
Before 2013, several families of representation coexisted. On the non-neural side, the main options were count-based vectors such as LSA, obtained by factorizing a co-occurrence matrix (Faruqui & Dyer 2014), and Brown clusters (Turian et al. 2010). Neural language models also learned vectors as a by-product. Notable examples are Collobert and Weston's (2008) ranking-loss model and Mnih and Hinton's (2009) HLBL, a hierarchical log-bilinear model (Turian et al. 2010). Turian and colleagues compared these representations under controlled conditions on named-entity recognition and chunking. Every representation improved accuracy. Brown clusters handled rare words better than the embeddings, which mattered most for NER (Turian et al. 2010).
Other models of the period:
- Maas et al. added a supervised sentiment term so that vectors also captured polarity (Maas et al. 2011).
- Huang et al. combined local window context with document context and learned multiple vectors per word for ambiguous words. They trained on Wikipedia text of about 990M tokens (Huang et al. 2012).
The turning point came in 2013. Vectors taken from a recurrent language model answered 39.6% of a new 8000-question syntactic analogy set. The strongest LSA model, built from identical training data, reached only 16.5% (Mikolov et al. 2013). The word2vec papers then introduced the simpler CBOW and skip-gram architectures, which could be trained cheaply on very large corpora (Mikolov et al. 2013). They also added negative sampling, frequent-word subsampling and phrase vectors (Mikolov et al. 2013). GloVe followed with a count-based global objective (Pennington et al. 2014).
A parallel line of work addressed morphology. Luong et al. built word vectors from morpheme vectors with a recursive network, with the largest gains on rare words (Luong et al. 2013). fastText later represented each word as a bag of character n-grams (Bojanowski et al. 2017). That approach was scaled to 157 languages using Wikipedia plus Common Crawl (Grave et al. 2018). It was also improved for English with phrases, position-dependent weights and deduplicated training on 630B words of Common Crawl (Mikolov et al. 2017).
| Approach | Year | Signal used |
|---|---|---|
| C&W / HLBL (compared by Turian et al., 2010) | 2008–2009 | language-model by-product |
| word2vec (CBOW, skip-gram) | 2013 | local context prediction |
| GloVe | 2014 | global co-occurrence counts |
| fastText | 2017 | context prediction plus subwords |
| ELMo | 2018 | deep bidirectional language model |
In 2018, contextualized representations such as ELMo became prominent. They assign each token a vector that depends on its sentence (Peters et al. 2018). Language models built with LSTM, Transformer and gated-CNN backbones all beat GloVe on tasks such as NLI, semantic role labeling, parsing and NER (Peters et al. 2018).
How it is used today
The most common use is as an input layer for task models. Pretrained vectors used as the only input let recurrent taggers outperform feature-rich CRFs on aspect extraction (Liu et al. 2015). Embeddings can also be specialized for a task. Sentiment-specific vectors were trained on about 10 million emoticon-labelled tweets. On SemEval 2013 positive/negative classification they reached 84.89% macro-F1, roughly matching the 84.70% of the top hand-engineered system, Mohammad et al. (2013) (Tang et al. 2014).
Cross-lingual mapping is a second major application. The usual tool is a linear map between monolingual spaces:
- For English-to-Spanish word translation with skip-gram vectors trained on WMT11 text, two changes raised P@1 from about 35% to about 41%. The changes were normalizing the vectors and restricting the map to be orthogonal; the 35% baseline is Mikolov's unconstrained linear map (Xing et al. 2015).
- A later framework adds orthogonality, length normalization and mean centering to the least-squares mapping objective, and recasts earlier methods as special cases. It yields an exact closed-form solution, W = VUᵀ, where UΣVᵀ is the SVD of ZᵀX and X and Z hold the source and target vectors of the dictionary pairs (Artetxe et al. 2016).
- CCA on translation pairs can even improve monolingual vectors (Faruqui & Dyer 2014).
- Bilingual embeddings have served as a phrase-pair similarity feature in phrase-based translation (Zou et al. 2013).
Supervision has since shrunk toward zero. Self-learning works from a seed of 25 word pairs or a list of numerals (Artetxe et al. 2017). A fully unsupervised method was tested on an extended version of the Dinu et al. dataset. It reached 33.50% English–Finnish translation accuracy in its best of ten runs and 32.63% on average, where earlier adversarial unsupervised methods stayed below 2% (Artetxe et al. 2018).
Embeddings also serve as measurement instruments. Decade-by-decade vectors aligned with orthogonal Procrustes revealed two regularities in semantic change. Frequent words change more slowly, and polysemous words change faster once frequency is controlled for (Hamilton et al. 2016). The MoverScore metric compares generated texts with references using Earth Mover's Distance over contextualized embeddings, chiefly BERT (Zhao et al. 2019). On the engineering side, product quantization shrinks fastText classifiers to roughly a hundredth of their memory with little loss in accuracy (Joulin et al. 2016).
Open questions
Evaluation remains unsettled. When six embedding methods were trained under matched conditions, their ranking changed with the criterion used. Downstream scores also proved an unreliable proxy for general quality (Schnabel et al. 2015). The same study found that vectors encode word frequency strongly enough to distort cosine similarity (Schnabel et al. 2015). Analogy accuracy, the classic benchmark, partly reflects choices in the scoring rule (Levy & Goldberg 2014).
Social bias is a second concern. Gender is largely captured by a single direction in word2vec space. Hard debiasing lowered the share of generated analogies that crowd workers judged stereotypical from about 19% to about 6% (Bolukbasi et al. 2016). For contextual models, standard cosine-based tests miss biases that probability-based probes reveal (Kurita et al. 2019).
Finally, the relation between static and contextual vectors is still being mapped. Static vectors conflate word senses, which motivated multi-prototype models (Huang et al. 2012). Probes show that contextual spaces encode dependency trees geometrically, as squared distances after a learned linear transformation (Hewitt & Manning 2019). Yet their advantage over word-level baselines is much smaller for semantic than for syntactic phenomena (Tenney et al. 2019). What exactly context adds beyond word identity remains an active research question.
Key papers
Related concepts
- Distributional semantics — Distributional semantics is the study and modelling of meaning through the statistics of linguistic context. Words, phrases or sentences…
- Semantic similarity — Semantic similarity is a graded measure of how close two linguistic units, such as words, concepts, sentences or documents, are in meaning.…
- Language model — A language model is a probability distribution over sequences of tokens (words, subwords or characters). It is usually factored into…
- Transformer and attention — The Transformer is a neural network architecture that represents a sequence by letting each element gather information from all others…
More papers
Topics: 📖 Lexical semantics & word embeddings · 🧠 Language models & pretraining
Shelves: 💬 Natural language processing
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/word-embeddings/