📄 Probabilistic Latent Semantic Analysis

Summary

Hofmann revisits Latent Semantic Analysis (LSA), which compresses term-document count tables with a truncated SVD. That approach works in practice but has a weak theoretical basis. He replaces it with Probabilistic Latent Semantic Analysis (PLSA), built on the aspect model. In this latent class model, each word-document co-occurrence is explained by a hidden topic variable z, and documents and words are conditionally independent given z. Parameters are fitted by maximum likelihood with EM. To limit overfitting, the paper introduces a tempered variant (TEM) controlled by an inverse temperature β and tuned on held-out data. Unlike LSA's squared-error fit, PLSA minimizes KL divergence to the empirical distribution. As a result, its factors are proper word distributions, and standard statistical tools can guide model selection. In retrieval experiments on four classic test collections, the resulting PLSI method clearly beat both LSI and plain term matching. The paper also argues that likelihood-based criteria such as perplexity can serve as a general yardstick for text analysis methods.

Key points

• Model and fitting. The aspect model has an asymmetric form, P(d)·Σ P(w|z)P(z|d), and an equivalent symmetric form, Σ P(z)P(d|z)P(w|z). It is fitted with EM, where the E-step computes P(z|d,w) and the M-step re-estimates counts weighted by those posteriors.
• Polysemy demo. On the CLUSTER dataset (1568 abstracts about clustering), a 128-factor model splits ambiguous words into separate topics:
  • "segment": image regions vs. speech
  • "matrix": linear algebra vs. manufacturing
  • "line": image lines vs. spectral lines
  • "power": astrophysics vs. VLSI
• Retrieval benchmarks. The test collections were MED (1033 abstracts), CRAN (1400), CACM (3204) and CISI (1460). PLSA models with K = 32, 48, 64, 80 and 128 were trained by TEM with 10% held-out data. LSI was tuned over 32–512 dimensions in steps of 8.
• Retrieval results.
  • Relative precision gains over the cosine baseline were often around 100% at intermediate recall levels.
  • PLSI still worked on collections where LSI failed completely.
  • PLSI*, which averages models with uniform weights, beat the best single model on every collection.
• Perplexity vs. precision. On MED, perplexity and average precision tracked each other closely as β varied.

Interesting ideas

• I1. Re-frame a heuristic decomposition as a likelihood model. Treating SVD-style factorization as a probabilistic mixture gives interpretable factors, normalized outputs and principled model selection. The same move could suit other count matrices, such as trade flows or region-by-feature tables.
• I2. Temper EM with a validation-tuned knob. Adding an inverse temperature β to EM and choosing it on held-out data is a cheap, general guard against overfitting. It applies to any latent variable model trained by EM.
• I3. Average models instead of picking K. Uniformly averaging similarity scores from models with different numbers of topics was robust and beat the best single model. It also removes the need to choose one dimensionality, which is useful wherever K is hard to set.
• I4. Fold in new items with fixed factors. Unseen queries or documents are mapped into topic space by keeping P(w|z) fixed and estimating only their mixing weights. This is a practical way to embed new items without retraining.
• I5. Use an intrinsic metric as a proxy for downstream quality. Perplexity correlated strongly with retrieval precision. This suggests unsupervised likelihood can steer model tuning when relevance labels are scarce, as is often the case for lower-resource languages like Ukrainian.