≈ Semantic similarity
Semantic similarity is a graded measure of how close two linguistic units, such as words, concepts, sentences or documents, are in meaning. It is computed either from structured knowledge such as taxonomies, or from vector representations, and is evaluated against human judgments.
What it is
Semantic similarity asks how much two expressions mean the same thing. Human raters typically use a bounded scale, such as the 0–4 scale of the Miller and Charles noun ratings (Resnik 1995). On such a scale, car and automobile score near the maximum. Car and bicycle share a good deal. Car and philosophy share almost nothing.
The output is a number on a continuous scale rather than a yes/no verdict. Being graded separates it from paraphrase detection, which gives a binary decision. Being symmetric separates it from textual entailment, which runs in one direction (Agirre et al. 2012).
A key distinction is between similarity and relatedness. Similarity in the narrow sense rests on shared properties and taxonomic closeness, as with cat and dog. Relatedness is a broader notion. It also covers part–whole links, antonymy and plain functional or associative connections, as with car and wheel, or coffee and cup (Budanitsky & Hirst 2006). Many benchmarks blur the two. In WS-353, for example, cup–coffee is rated higher than car–train (Faruqui et al. 2016). Some datasets set out to measure relatedness on purpose, such as the sentence-level SICK corpus (Marelli et al. 2014).
The key idea
There are two broad families of methods: symbolic (knowledge-based) and embedding (distributional) methods.
Symbolic methods place concepts in a hand-built taxonomy such as WordNet and derive similarity from its structure. Embedding methods map words or sentences to vectors learned from text and compare those vectors geometrically. The standard geometric comparison is cosine similarity:
cos(u, v) = (u · v) / (‖u‖ ‖v‖)
Here u and v are the two embedding vectors, u · v is their dot product, and ‖u‖ and ‖v‖ are their Euclidean lengths. The result ranges from −1 to 1 and ignores vector magnitude, so only direction counts.
For taxonomies, three classic measures stand out: Wu–Palmer, Resnik and Lin. The second line below is not a measure. It defines information content, which the last two measures build on:
sim_WP(c1, c2) = 2 · depth(lcs) / (depth(c1) + depth(c2))
IC(c) = −log p(c)
sim_Res(c1, c2) = IC(lcs(c1, c2))
sim_Lin(c1, c2) = 2 · IC(lcs) / (IC(c1) + IC(c2))
In these formulas, c1 and c2 are concepts. The term lcs(c1, c2) is their lowest common subsumer, the most specific concept that is an ancestor of both. The function depth gives a node's distance from the taxonomy root. The term p(c) is the probability of meeting an instance of c, or of any of its descendants, in a corpus, and IC(c) is the information content of c.
The Wu–Palmer measure uses depth alone (Corley & Mihalcea 2005). Resnik's measure scores a pair by how informative their shared ancestor is. For words, it takes the maximum over all sense pairs (Resnik 1995). Lin's measure normalizes that shared information by the information content of the two concepts themselves, which yields a value between 0 and 1 (Budanitsky & Hirst 2006).
Word-level measures can be lifted to sentences in several ways:
- Match each word to its most similar counterpart in the other text, weight the matches by inverse document frequency, and average the scores in both directions (Corley & Mihalcea 2005).
- Encode each sentence as a single vector and take the cosine between the two vectors (Wieting et al. 2015).
History and founding papers
One early line of computational work built on taxonomies. Resnik argued that simply counting edges treats every taxonomy link as an equal distance, even though links in WordNet span very uneven amounts of meaning. He therefore attached corpus probabilities to the nodes. Miller and Charles (1991) had rated 30 noun pairs, and WordNet can score 28 of them because one noun is missing. On those pairs, his information-content measure correlated with human ratings at r = 0.79, against 0.66 for edge counting. For comparison, the average per-subject correlation in his replication of the human study was about 0.88 over all 30 pairs and about 0.90 over the 28 (Resnik 1995).
Dictionary glosses offered another route. Extended gloss overlaps compare WordNet definitions of the two concepts and of their linked concepts. The measure correlates with human relatedness judgments at 0.6 or above, and it works for verbs and adjectives as well as nouns (Banerjee & Pedersen 2003).
Budanitsky and Hirst compared five WordNet measures on a practical task: detecting and correcting malapropisms. These are real-word spelling errors, where the mistyped form is itself a valid word that fits its context poorly. The best performer was the Jiang–Conrath measure, an information-content method that treats IC(c1) + IC(c2) − 2·IC(lcs) as a distance between the concepts. The authors also argued that distributional similarity does not substitute well for relatedness (Budanitsky & Hirst 2006).
Sentence-level evaluation was formalized by the SemEval Semantic Textual Similarity (STS) series. The 2012 pilot asked crowd workers to rate sentence pairs on a 0–5 scale. It drew 35 teams and 88 runs. The best systems exceeded 0.80 Pearson correlation, while a lexical baseline reached 0.31 (Agirre et al. 2012). Later editions extended the series in several directions:
| Edition | New direction |
|---|---|
| 2013 | Typed similarity pilot (Agirre et al. 2013) |
| 2014 | Spanish data (Agirre et al. 2014) |
| 2015 | Interpretable STS pilot (Agirre et al. 2015) |
| 2016 | Cross-lingual Spanish–English pilot (Agirre et al. 2016) |
| 2017 | Multilingual and crosslingual focus (Cer et al. 2017) |
Early winning STS systems were hybrids. The 2012 system that ranked first on two of the three official metrics combined over 300 feature variants, including Resnik-based WordNet similarity, in a regression model (Bär et al. 2012). The 2013 run with the highest mean correlation used latent semantic analysis (LSA), which compresses word co-occurrence counts into a few hundred dimensions. It paired these LSA word vectors with WordNet path distances and a simple word alignment (Han et al. 2013).
Neural models then took over. Siamese CNNs compared sentences from multiple perspectives (He et al. 2015). Pairwise word-interaction networks compared every word in one sentence with every word in the other (He & Lin 2016). A notable finding was that simple averaged paraphrastic word vectors transfer across domains better than LSTMs (Wieting et al. 2015). Scaling this approach with back-translated paraphrase data produced embeddings that beat every SemEval STS submission from 2012 to 2016 (Wieting & Gimpel 2018).
How it is used today
Most current practice encodes sentences independently and compares them by cosine. Sentence-BERT fine-tunes BERT in a siamese setup so that cosine distances become meaningful (Reimers & Gurevych 2019).
Raw BERT sentence vectors, averaged over upper layers, turned out to be anisotropic. They crowd into a narrow region instead of spreading evenly in all directions, in a layout shaped by word frequency. A normalizing flow is an invertible learned transformation, trained here without labels. It maps these vectors to an isotropic Gaussian. Compared with the same frozen BERT vectors without the flow, this raised scores on seven STS tasks by 8.16 Spearman points on average (Li et al. 2020).
Contrastive training now dominates:
- SimCSE has two variants. Unsupervised SimCSE passes each sentence through the encoder twice, and the two outputs, made slightly different by dropout, form a positive pair. Supervised SimCSE takes entailment pairs from natural language inference (NLI) data as positives and contradictions as hard negatives. With BERT-base, the two variants reach average Spearman correlations of 76.3 and 81.6 across seven STS tasks (Gao et al. 2021).
- DiffCSE adds replaced-token detection: a discriminator must spot which tokens a small generator has swapped into the sentence. This improves on unsupervised SimCSE by about 2.3 points on STS (Chuang et al. 2022).
Not every useful encoder is contrastive. Sent2Vec is an earlier and cheaper model that averages learned word and n-gram vectors. It remains competitive where compute is limited (Pagliardini et al. 2018).
Multilingual similarity is a major application area. LASER embeds 93 languages in one shared space (Artetxe & Schwenk 2018). LaBSE is trained on translation pairs. Its STS-B test Pearson of about 72.8 falls below models built for English, which suggests that translation-oriented training does not maximize English graded similarity (Feng et al. 2020). Knowledge distillation can transfer an English similarity space to many other languages with little language bias (Reimers & Gurevych 2020).
Applications include retrieval, paraphrase mining, machine translation evaluation, question answering and short-answer grading. In grading, WordNet and LSA similarities feed alignment-based models (Mohler et al. 2011).
At the word level, embeddings are still routinely scored on SimLex-999 and WS-353. The score is the Spearman correlation between cosine scores and human ratings (Faruqui et al. 2016). SimLex was built to target similarity rather than relatedness (Faruqui et al. 2016).
Open questions
- Definition. What counts as "similar" remains underspecified. Besides the similarity–relatedness blur described above, benchmarks ignore polysemy and are small. Ten common word-similarity sets average 781 pairs, which makes significance testing and fixed data splits hard (Faruqui et al. 2016).
- Uncatalogued relations. Much perceived relatedness is "non-classical" or formed ad hoc in context. In Morris's experiments with human readers, about 60% of relations noticed in text fell into this category, and no resource lists them (Budanitsky & Hirst 2006).
- Sense confusion. Symbolic measures can match words through the wrong sense. For example, horse has a slang sense meaning heroin, which places it with tobacco under narcotics, so the two score as similar (Resnik 1995).
- Surface overlap. Embedding similarity can track word overlap too closely (Li et al. 2020).
- Unclear content of vectors. It is still unclear what a single sentence vector actually encodes, and probing studies try to answer this (Adi et al. 2016; Conneau et al. 2018).
- Dataset artifacts. SICK built contradiction pairs by rewriting source sentences. A participating team found that negation words alone identify 86.4% of contradiction pairs, a construction artifact that future releases should control for (Marelli et al. 2014).
- Interpretability. The 2015 interpretable STS pilot asked systems to align chunks across the two sentences, label each alignment and score it. Chunks are short phrase segments. Systems did clearly worse when they had to find the chunks themselves than when gold chunks were given (Agirre et al. 2015).
- Single-score evaluation. Correlation with one human score may not capture the many dimensions along which meanings can match (Agirre et al. 2013).
Key papers
Related concepts
- Distributional semantics — Distributional semantics is the study and modelling of meaning through the statistics of linguistic context. Words, phrases or sentences…
- Word embeddings — Word embeddings are dense, real-valued vectors that represent the words of a vocabulary as points in a shared geometric space. They are…
- Lexical semantics — Lexical semantics is the study of word meaning. It asks what words and multiword expressions denote, how a word's senses are organized, how…
- Word sense disambiguation — Word sense disambiguation (WSD) identifies which meaning of an ambiguous word is intended in a given context. It is usually framed as…
More papers
Topics: 📖 Lexical semantics & word embeddings
Shelves: 💬 Natural language processing
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/semantic-similarity/