🧊 Latent semantic indexing
Latent semantic indexing (LSI) is a retrieval and text-analysis method that factorizes a weighted term–document matrix with a truncated singular value decomposition (SVD). Words and documents are then compared in a small set of latent dimensions instead of by exact word overlap.
What it is
A search engine that matches only literal words misses a document about "automobiles" when the user types "car". Latent semantic indexing addresses this gap. It reads a whole collection of texts and learns which words tend to appear in similar documents. It then places words and documents as points in a compact space, so that texts about the same subject sit close together even when they share few words. The name LSI is usual in information retrieval. The same technique is called latent semantic analysis (LSA) when it is used to model word meaning.
LSI is one of the earliest working implementations of distributional semantics, the view that a word's meaning shows in the company it keeps. It is also an ancestor of topic modeling and of dense word embeddings.
Both acronyms are overloaded. In clinical linguistics "LSA" also means language sample analysis, a diagnostic procedure for children's speech (Ryser et al. 2026). In legal natural language processing "LSI" can mean legal statute identification (Paul et al. 2026).
The key idea
The term–document matrix
The starting point is a matrix X with one row per term and one column per document. Each cell holds how often the term occurs in the document. The counts are usually reweighted, for example with term frequency–inverse document frequency (TF-IDF) or log-entropy weighting, so that very frequent function words do not dominate. Cross-lingual LSA systems, for instance, have used log-entropy weighting on verse-aligned Bible translations (Chew & Abdelali 2008). The matrix is mostly zeros, the same sparsity that an inverted index exploits.
Truncated SVD
X ≈ U_k Σ_k V_kᵀ
The symbols mean the following:
- X is the t × d weighted term–document matrix, for t terms and d documents.
- k is the chosen number of latent dimensions, typically a few hundred.
- U_k is a t × k matrix with orthonormal columns. Row i of U_k describes term i.
- Σ_k is a k × k diagonal matrix holding the k largest singular values. Each squared singular value, divided by the squared Frobenius norm of X, gives the share of X that its dimension captures.
- V_k is a d × k matrix with orthonormal columns. Row j of V_k describes document j.
- ᵀ denotes transposition.
Keeping only the top k singular triples gives the best rank-k approximation of X in the least-squares sense. A singular triple is a singular value together with its left and right singular vectors. Similarity between items is then measured by the cosine between their reduced vectors.
Why synonymy is handled, and polysemy only partly
Suppose two synonyms appear in documents that share the same other words. The rank reduction cannot afford separate dimensions for them, so it merges them, and their vectors end up close. A query containing one synonym therefore retrieves documents that contain only the other.
Polysemy is handled less well. Each word receives a single vector, so a word with two senses is placed at a blend of its contexts. Context still helps in retrieval. A query or document vector is a weighted sum of its term vectors, so an ambiguous term is pulled toward the sense suggested by the terms around it. Explicitly separating senses requires other machinery, as in word sense disambiguation.
Folding in queries
A new query or document is not re-decomposed. Its term vector q is projected into the existing space:
q̂ = Σ_k⁻¹ U_kᵀ q
Here q is the weighted term vector of the query, U_kᵀ maps it onto the k latent term directions, and Σ_k⁻¹ rescales each direction by the inverse of its singular value. The result q̂ is comparable with the rows of V_k. Multilingual LSA projects unseen documents with the same projection (Chew & Abdelali 2008). Folding in is cheap, but the space drifts out of date as a collection grows, and periodic recomputation is then needed.
History and founding papers
The method was introduced by Deerwester, Dumais, Furnas, Landauer and Harshman in 1990 for document retrieval. Landauer and Dumais (1997) later recast LSA as a theory of how people acquire word meaning from exposure to text. This made the method a reference point in psycholinguistics and in models of semantic similarity.
The main criticism was theoretical. A least-squares fit to count data has no clear probabilistic meaning, and its latent dimensions can take negative values. Hofmann's probabilistic latent semantic analysis (pLSA) replaced the SVD with an aspect model. In that model a hidden topic explains each word–document co-occurrence. The parameters are fitted by expectation–maximization (EM), with a tempered variant that limits overfitting (Hofmann 1999).
On four classic test collections (MED, CRAN, CACM and CISI), retrieval with pLSA clearly outperformed LSI. Its relative precision gains over a cosine term-matching baseline were often close to 100% at intermediate recall levels. It also kept working on collections where LSI failed outright (Hofmann 1999). Its factors separated senses of ambiguous words such as "segment" (image regions versus speech) (Hofmann 1999).
Blei, Ng and Jordan then argued that pLSA lacks a generative account at the document level. Its parameters grow with the corpus, and it cannot score unseen documents in a principled way. Latent Dirichlet allocation fixed this by drawing each document's topic proportions from a Dirichlet prior (Blei et al. 2003).
| Model | Factorization | Fitting | New documents |
|---|---|---|---|
| LSI/LSA | truncated SVD of weighted counts | least squares | folding in |
| pLSA | aspect model P(w|z)P(z|d) | EM, minimizing Kullback–Leibler divergence | folding in via EM (no generative model of documents) |
| LDA | Dirichlet–multinomial hierarchy | variational EM | proper posterior |
How it is used today
As a lexical-semantic tool
In the 2000s LSA served as a cheap, unsupervised meaning space for many tasks.
- Dialogue acts: Feature LSA added speaker and dialogue-game values as pseudo-words before the SVD. Against a majority baseline, plain LSA reduced errors by a third to a half, and the feature-augmented version reduced them further (Serafin & Eugenio 2004).
- Idioms: comparing the LSA vector of a German phrase with the sum of its parts' vectors gave a compositionality score. This score separated idiomatic from literal multiword expressions (Katz & Giesbrecht 2006).
- Verb classes: LSA vectors helped induce verb classes for Modern Standard Arabic (Snider & Diab 2006).
- Spoken-response checking: a k-nearest-neighbours (KNN) classifier over LSA topic spaces was the baseline for detecting off-topic answers in spoken English tests (Malinin et al. 2016).
- Essay scoring: Arabic essay scorers once relied on LSA, which requires training essays on the same topic (Alqahtani & Alsaif 2020).
Against word embeddings
The GloVe paper framed the field as a choice between two families. Global matrix factorizations such as LSA use corpus statistics well but do poorly on analogies. Local-window predictors handle analogies better. GloVe was proposed as a synthesis of the two (Pennington et al. 2014).
Later comparisons used SVD over positive pointwise mutual information (PPMI) counts alongside skip-gram as standard representations (Levy et al. 2015). SVD-based spaces trained on the same text can be aligned to skip-gram spaces with supervision, yet unsupervised alignment between them fails entirely (Hartmann et al. 2018). Historical studies of language change still report SVD embeddings next to word2vec (Luo et al. 2019). In direct contests the predictive models often win. For Arabic real-word spelling errors, for example, Skip-Gram beat LSA on precision and recall (Ben Othmane Zribi 2018).
The SVD lives on inside large models
The same truncation idea now operates on network weights rather than term counts, mainly in large language models.
- Compression: low-rank compression chooses per-layer ranks by learned masks over singular values (Sundrani et al. 2026). Other methods keep error-prone columns dense (Xv et al. 2026). Some apply principal component analysis (PCA) to activations instead of weights (Tian et al. 2026). Variance-based truncation can discard low-variance but loss-sensitive directions that carry factual knowledge (Shihab et al. 2026).
- Adaptation: adaptive variants of low-rank adaptation (LoRA), such as AdaLoRA, express updates in SVD form and prune singular values. HyperAdaLoRA speeds up this rank allocation with hypernetworks (Zhang et al. 2026). Other variants learn which spectral band to train (Zhou et al. 2026), initialize from residual singular vectors (Luo et al. 2026), or clip singular values to reduce forgetting (Wi & Park 2026).
- Merging and editing: shared SVD bases are used to align merged adapters (Lu et al. 2026). In knowledge editing, edits are steered away from high-energy singular directions (Zhang et al. 2026).
- Analysis: SVD also serves as a diagnostic. One example is the nuclear norms (sums of singular values) of layer gradients (Li et al. 2025). Another is the joint SVD of per-language gradients in speech translation (SUN et al. 2026).
- Related work: SVD-based substitutes for the output softmax exist (Milakov & Gimelshein 2018), and small transformers can learn to compute SVDs themselves (Charton 2021).
Open questions
- Choosing k. Retrieval quality depends on the number of dimensions, and tuning it over a wide range was standard practice (Hofmann 1999). Principled criteria, such as minimum description length or likelihood, remain contested for SVD.
- Interpretability. Latent axes are linear mixtures with signed weights, unlike pLSA's word distributions (Hofmann 1999). What, if anything, an individual dimension means is unsettled.
- Cross-lingual spaces. Language relatedness and morphological preprocessing change multilingual LSA results (Chew & Abdelali 2008). How to build fair shared spaces for typologically distant languages is still open.
- Relational meaning. Vectors of single words may simply not encode relations such as hypernymy (Levy et al. 2015). In those tests the finding held for count-based and predictive vectors alike, but only supervised lexical-inference classifiers were examined (Levy et al. 2015).
- What to keep. In compression and adaptation, the largest singular directions are not always the most important ones (Shihab et al. 2026; Wi & Park 2026). This echoes the old question of whether variance is the right criterion for meaning.
Key papers
Related concepts
- TF-IDF — Term frequency–inverse document frequency (TF-IDF) is a weighting scheme that scores how important a word is to one document in a…
- Topic modeling — Topic modeling is a family of mostly unsupervised methods that discover recurring themes in a document collection. Each theme, or topic, is…
- Distributional semantics — Distributional semantics is the study and modelling of meaning through the statistics of linguistic context. Words, phrases or sentences…
- Word embeddings — Word embeddings are dense, real-valued vectors that represent the words of a vocabulary as points in a shared geometric space. They are…
More papers
Topics: 🔍 Information retrieval & search · 📖 Lexical semantics & word embeddings
Shelves: 🤖 Artificial Intelligence
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/latent-semantic-indexing/