🗂️ Topic modeling

About 18 minutes to read · every paper named here has its own page in this library

What it is

Suppose you have ten thousand newspaper articles and no time to read them. A topic model reads them for you, in a statistical sense. It returns a short list of themes, each shown as its most characteristic words: one might be "court, judge, trial, appeal", another "match, goal, season, coach". It also tells you how much of each theme every article contains. Nobody tells the model in advance what the themes are. They come out of the patterns of words that tend to appear together across documents.

That basic promise has made topic modeling one of the most widely used tools of text mining and a standard entry point into natural language processing (NLP) for researchers outside computer science. Historians, political scientists and media scholars use it to get a first map of a large corpus. Engineers use topic vectors as compact document features.

Three ideas recur throughout the field:

• A topic is a probability distribution over a vocabulary. Its "top words" are just the words with the highest probability.
• A document is not assigned to a single topic. It is treated as a mixture, so an article about a court case over a football transfer can be part law and part sport.
• The model is fitted without labels. This makes it exploratory, and also makes judging whether its output is "good" a research problem in its own right.

Topic modeling draws on distributional semantics, the idea that words occurring in similar contexts have related meanings. It now overlaps with word embeddings and large language models (LLMs).

How it works

The generative story of LDA

The canonical model is latent Dirichlet allocation (LDA), a three-level hierarchical Bayesian model (Blei et al. 2003). Like most probabilistic topic models, it is defined by an imagined process that could have produced the corpus. Fitting the model means running that story backwards.

Assume K topics, each a distribution β_k over a vocabulary of V word types. For every document d:

θ_d   ~ Dir(α)                 (topic proportions of document d)
z_dn  ~ Mult(θ_d)              (topic of the n-th word position)
w_dn  ~ Mult(β_{z_dn})         (the word itself, drawn from that topic)

Each symbol has a specific role:

• θ_d is a vector of K non-negative numbers summing to one, giving the share of each topic in document d.
• Dir(α) is the Dirichlet distribution with concentration parameter α. Small values of α favour documents dominated by a few topics.
• z_dn is a hidden label saying which topic produced position n of document d.
• Mult denotes a single draw from a categorical (multinomial) distribution.
• β_{z_dn} is the word distribution of the chosen topic, one row of a K×V matrix.
• w_dn is the observed word.

The original formulation also draws a document length from a Poisson distribution, which the authors note is not essential. It treats K as known and fixed (Blei et al. 2003).

The Dirichlet was chosen for convenience as much as realism. It belongs to the exponential family, has finite-dimensional sufficient statistics, and is conjugate to the multinomial, which simplifies inference (Blei et al. 2003). The model ignores word order (the "bag of words" assumption) and treats documents as exchangeable. Only co-occurrence within documents matters.

The decisive difference from its predecessor, probabilistic latent semantic analysis (pLSA), is that θ_d is a random variable with a prior rather than a free parameter per document. In pLSA the parameter count grows linearly with the number of documents, which encourages overfitting and leaves no principled way to score a new text. LDA assigns a proper probability to unseen documents (Blei et al. 2003).

Inference

The posterior over θ, z and β given the words cannot be computed exactly. Two families of approximation dominate.

Variational inference replaces the true posterior with a simpler factorised distribution and tunes it to be as close as possible. Blei, Ng and Jordan derived such an approximation and wrapped it in a variational EM procedure that also estimates α and β by empirical Bayes (Blei et al. 2003). The same machinery extends to models without conjugacy. The dynamic topic model, for instance, uses variational parameters as pseudo-observations smoothed by a Kalman filter or by wavelet regression (Blei & Lafferty 2006). Many neural topic models use amortised variational inference, in which a neural encoder predicts each document's topic proportions directly. The embedded topic model is one example, and this design lets it scale to large collections (Dieng et al. 2020).

Gibbs sampling, a Markov chain Monte Carlo method, takes a different route. In the collapsed form popularised by Griffiths and Steyvers (2004), θ and β are integrated out. The sampler repeatedly reassigns each word token's topic z_dn given all other assignments. Each reassignment needs only counts of how often the word appears in each topic and how often each topic appears in the document. Collapsed Gibbs sampling is the workhorse of many LDA variants:

• topic coherence studies (Newman et al. 2010);
• supervised Labeled LDA (Ramage et al. 2009);
• selectional-preference models (Ritter et al. 2010; Ó Séaghdha 2010);
• the polylingual topic model, which also re-estimates hyperparameters during sampling (Mimno et al. 2009).

Choosing the number of topics

K is the most consequential setting, and LDA does not learn it (Blei et al. 2003). In practice it is chosen in one of several ways:

• by held-out likelihood, typically reported as perplexity;
• by coherence scores computed over a range of K;
• by task-specific validation, as in a cluster-consistency procedure used to pick a small number of review aspects (Brody & Elhadad 2010);
• by human inspection.

Nonparametric Bayesian models such as the hierarchical Dirichlet process (Teh et al., 2006) and criteria related to minimum description length offer more principled alternatives. Clustering-based methods sidestep the choice altogether. Top2Vec takes the number of dense regions in a joint document–word embedding space as the number of topics (Angelov 2020). CobwebTM grows a topic hierarchy whose depth and width are set by the data (Singaravadivelan et al. 2026). In authorship attribution with topic models, the number of topics had the largest effect on performance, and inferring it automatically was left as future work (Seroussi et al. 2014).

Perplexity and coherence

The classic intrinsic measure is perplexity on held-out text:

perplexity(D_test) = exp( − Σ_d log p(w_d) / Σ_d N_d )

Here D_test is a set of held-out documents, w_d the words of document d, p(w_d) the probability the trained model assigns to them, and N_d the number of words in d. Lower is better. Hofmann proposed likelihood-based criteria of this kind as a general yardstick for text analysis (Hofmann 1999). LDA was shown to achieve lower held-out perplexity than pLSA and a mixture of unigrams (Blei et al. 2003).

Perplexity, however, says nothing about whether people can read the topics. Coherence measures try to capture that directly. A widely used variant averages normalised pointwise mutual information (NPMI) over pairs of a topic's top words:

NPMI(w_i, w_j) = log[ P(w_i, w_j) / (P(w_i) · P(w_j)) ] / ( − log P(w_i, w_j) )

Here P(w_i) is the probability that word w_i appears in a document or sliding window of a reference corpus, and P(w_i, w_j) is the probability that both appear together. The score ranges from −1 to 1, with higher values meaning the words co-occur more than chance. The section on evaluation below traces how these measures were developed and validated.

History

From matrices to probabilities

The lineage begins with latent semantic analysis (LSA), known in retrieval as latent semantic indexing (LSI) (Deerwester et al., 1990). LSA builds a term–document count matrix and compresses it with a truncated singular value decomposition (SVD), so that documents and words live in a low-dimensional "semantic" space. It worked well enough to become standard in information retrieval, but its squared-error objective lacked a clear statistical meaning.

Thomas Hofmann's pLSA (1999), called probabilistic latent semantic indexing (pLSI) when used for retrieval, recast the idea as a latent class "aspect" model:

P(d, w) = P(d) · Σ_z P(w | z) · P(z | d)

Here d is a document, w a word, z a hidden aspect (topic), P(w | z) the topic's word distribution and P(z | d) the document's topic weights. Words and documents are independent given z. The parameters are fitted by expectation–maximisation, with a tempered variant to curb overfitting (Hofmann 1999). Because pLSA minimises a KL divergence rather than squared error, its factors are genuine word distributions. On four classic retrieval collections it outperformed both LSI and plain term matching. It also separated senses of ambiguous words such as "segment" and "power" into different factors (Hofmann 1999).

LDA added a Dirichlet prior over document mixtures and became the field's most influential model (Blei et al. 2003). LSA did not disappear. It continued to serve tasks such as grading the compositionality of German multiword expressions (Katz & Giesbrecht 2006), cross-language retrieval trained on verse-aligned Bible translations (Chew & Abdelali 2008), and dialogue-act classification with extra annotation features injected into the matrix (Serafin & Eugenio 2004).

Structure, time and supervision

From the mid-2000s to the end of the decade, the field explored LDA as a modular building block. Dynamic topic models let each topic drift across time slices of a corpus (Blei & Lafferty 2006). The correlated topic model (CTM) of Blei and Lafferty replaced the Dirichlet with a logistic normal so that topics could co-occur in correlated ways. Supervised variants tied topics to labels (Ramage et al. 2009), and multilingual variants linked topics across languages (Mimno et al. 2009).

The evaluation turn

Around 2009 a strand of work asked whether topics actually make sense to people. Human studies found that better predictive likelihood often went together with less interpretable topics (Chang et al. 2009). This triggered half a decade of research on automatic coherence (Newman et al. 2010; Mimno et al. 2011; Aletras & Stevenson 2013; Lau et al. 2014; Röder et al. 2015).

Embeddings, neural models and LLMs

From 2015 onward, pretrained word vectors entered topic models (Nguyen et al. 2015). Topics and words were then placed in a shared embedding space (Dieng et al. 2020; Angelov 2020). Contextual sentence embeddings from Transformer encoders were later fed into neural models (Bianchi et al. 2021) or clustered directly (Grootendorst 2022).

More recently, LLMs have been prompted to generate topics and assign them to documents (Pham et al. 2024). They have also been used to refine and evaluate the output of conventional models (Xu et al. 2026). A recent survey describes a shift from algorithm-centric pipelines to LLM-assisted and LLM-centric topic analysis (Xu et al. 2026).

Methods

Matrix factorisation

LSA and its relatives factor a weighted term–document matrix. Variants change what goes into the matrix rather than the factorisation itself:

• Feature LSA adds pseudo-word rows for categorical features such as speaker or dialogue game. It was evaluated on dialogue-act classification in Spanish CallHome, MapTask and a small tutoring corpus. Measured as relative error reduction over a majority-class baseline, plain LSA reached 33–52% and Feature LSA 60–78% (Serafin & Eugenio 2004).
• Multilingual LSA concatenates aligned translations, so documents in different languages share one space (Chew & Abdelali 2008).

Non-negative matrix factorisation (Lee and Seung, 1999) is a common modern counterpart. Recent work reuses vanilla LDA and NMF unchanged as binary relevance filters for German news on extreme climate events, with relevant topics identified automatically from hazard keywords (Madureira et al. 2026).

Bayesian extensions of LDA

Because LDA is a generative module, it has been specialised in many directions:

The dynamic model's key equation is a random walk:

β_{t,k} ~ N(β_{t−1,k}, σ²I)

Here β_{t,k} is the natural-parameter vector of topic k in time slice t, N a Gaussian, σ² the drift variance and I the identity matrix. The mean of the topic-proportion prior evolves the same way (Blei & Lafferty 2006). Applied to the journal Science from 1880 to 2000, the model predicted next-year documents better than static LDA. It also traced how word probabilities within topics shifted over time (Blei & Lafferty 2006), a concern it shares with research on language change. Recent dynamic neural models add optimal-transport matching between slices, so that topics can split and merge. They combine this with distillation from pretrained models and LLM refinement (Nguyen et al. 2026).

The "documents and words" abstraction also transfers to linguistic problems:

• In word sense disambiguation and sense induction, each target word gets its own small model in which context windows mix a handful of latent senses (Brody & Lapata 2009).
• For selectional preferences, a predicate is treated as a "document" whose "words" are its arguments, so topics become semantic classes (Ritter et al. 2010; Ó Séaghdha 2010).
• In sentiment and emotion analysis, sentence-level LDA recovers rateable aspects of restaurant reviews (Brody & Elhadad 2010). A hierarchical supervised variant combines topics with word polarity to detect sarcasm in tweets (Joshi et al. 2016).

Embedding-based and neural models

Embeddings address LDA's weakness on small corpora and short texts. LF-LDA and LF-DMM mix the usual Dirichlet-multinomial word draw with a component scored by the dot product of a topic vector and fixed pretrained word vectors. They improved coherence and short-text classification (Nguyen et al. 2015).

The embedded topic model (ETM) defines each topic's word distribution as a softmax over inner products between a topic vector and all word vectors. It remained stable as the vocabulary grew, so the rare words that practitioners usually prune can be kept. This matters because word frequencies follow Zipf's law, which puts most of the vocabulary in a long tail. The ETM also gathered stop words into a few dedicated topics (Dieng et al. 2020).

Attention-based aspect extraction offers a neural alternative to LDA-style aspect models. It replaces the topic model with an autoencoder over word embeddings that reconstructs sentences from aspect vectors (He et al. 2017).

Variational autoencoder topic models (ProdLDA, Srivastava and Sutton, 2017) became the base for contextualised models. CombinedTM concatenates an SBERT document embedding with the bag of words at the encoder input. It produced the most coherent topics on four of five English corpora (Bianchi et al. 2021). ZeroShotTM drops the bag of words from the input altogether. A model trained on English can then assign topics to Italian, French, Portuguese or German documents it never saw (Bianchi et al. 2021). Swapping in other encoders, such as a Doc2Vec–RoBERTa hybrid for classical Arabic Hadith, follows the same template (Aftar et al. 2024).

Clustering pipelines: Top2Vec and BERTopic

A second neural family abandons the generative story. Top2Vec embeds documents and words jointly with doc2vec, finds dense regions of document vectors, and describes each region by the words nearest its centroid (Angelov 2020).

BERTopic chains four steps:

1. Embed documents with Sentence-BERT.
2. Reduce dimensionality with UMAP.
3. Cluster with HDBSCAN, setting noisy documents aside as outliers.
4. Describe each cluster with a class-based TF-IDF (Grootendorst 2022).

The weighting in the last step is:

W(t, c) = tf(t, c) · log(1 + A / tf(t))

Here tf(t, c) is the frequency of term t in the concatenated text of cluster c, tf(t) its frequency across all clusters, and A the average number of words per cluster. Because the steps are decoupled, the embedding model can be swapped and topics over time can be computed without re-clustering. The method assumes each document carries one topic, however (Grootendorst 2022). Successors keep the c-TF-IDF labelling step while replacing the clustering: CobwebTM builds an incremental hierarchy of Gaussian nodes suited to streaming data (Singaravadivelan et al. 2026). Other work treats outliers as possible early signals of emerging topics rather than as noise (Zve et al. 2026).

LLM-based topic analysis

TopicGPT uses prompt engineering to make an LLM propose labelled topics with descriptions and merge near-duplicates. It then assigns topics to documents with supporting quotes. On Wikipedia it reached a harmonic-mean purity of 0.74 against 0.64 for the best baseline (Pham et al. 2024). LLMs also act as post-hoc refiners. LLM-XTM keeps only words that recur across several sampled rewrites of a cross-lingual topic's word list (Xuan et al. 2026). Lightweight graph methods offer a cheaper route to topic labels (Mekaoui et al. 2026).

Mechanistic topic models replace words with sparse-autoencoder features of LLM activations. The resulting topics can steer text generation (Zheng et al. 2026). In the other direction, in-context learning in LLMs has been analysed as inference of a latent concept, much as a topic model infers a latent topic (Wang et al. 2023). Segment-based topic allocation assigns topics to short spans rather than whole documents, to avoid contamination between themes (Yoon et al. 2026).

Evaluation methods

Chang and colleagues introduced two human tasks (Chang et al. 2009):

• word intrusion, in which annotators look for an out-of-place word inserted among a topic's top words;
• topic intrusion, in which they look for an out-of-place topic among a document's top topics.

Across pLSI, LDA and the correlated topic model, the CTM usually achieved the best predictive likelihood but the worst human scores.

Several automatic measures followed:

• Newman and colleagues set up coherence evaluation as a task. Annotators rated how coherent each topic's ten most probable words were, and automatic scorers built on WordNet, Wikipedia and Google were compared with those ratings. PMI computed from Wikipedia co-occurrence agreed with human ratings about as well as annotators agreed with one another (Newman et al. 2010).
• Mimno and colleagues proposed a score using only document co-occurrence within the training corpus. It was validated against expert ratings of NIH grant topics (Mimno et al. 2011).
• Aletras and Stevenson used distributional context vectors, which correlated better with crowdsourced ratings than averaged PMI (Aletras & Stevenson 2013).
• Lau, Newman and Baldwin showed that NPMI removes PMI's bias toward rare words, and they automated word intrusion (Lau et al. 2014).
• Röder, Both and Hinneburg decomposed coherence into segmentation, probability estimation, confirmation measure and aggregation. Searching this space produced the C_V measure (Röder et al. 2015).

Topic diversity, the share of unique words among the top 25 words of all topics, is commonly reported alongside coherence. Multiplying the two gives a combined quality score (Dieng et al. 2020).

Datasets and benchmarks

No single leaderboard governs topic modeling. Researchers reuse a handful of corpora.

Human-judgement datasets are the real benchmarks for coherence measures:

• The NYT and Wikipedia annotations of Chang et al. were reused and extended by later work (Lau et al. 2014).
• Aletras and Stevenson released ratings for 300 LDA topics drawn from news, newsgroups and genomics abstracts (Aletras & Stevenson 2013).
• Röder and colleagues pooled all public rating sets into six benchmarks and released the Palmetto toolkit (Röder et al. 2015).
• A recent study of philosophy-of-science journals adds a "topic word mixing" task that judges whether a word set blends two topics. It finds that co-occurrence coherence and word intrusion can disagree in specialised domains (Prouteau et al. 2026).

Other resources target particular settings:

• reviews with segment-level topic annotations (Yoon et al. 2026);
• a large classical Arabic exegesis annotated for topics and named entities (Ahmed et al. 2022);
• a Wikipedia corpus segmented by tables of contents, built for the related task of topical text segmentation, a problem studied in discourse analysis (Koshorek et al. 2018).

Uses in the humanities and social sciences

Much of the method's impact lies outside NLP. Digital humanities projects show topics directly to readers, which motivated the work on automatic interpretability (Lau et al. 2014). Religious and historical texts are a recurring target, including Hadith collections (Aftar et al. 2024) and Quranic exegesis (Ahmed et al. 2022). These studies pay particular attention to Semitic languages and their rich morphology.

Social scientists use topic models for framing and agenda analysis:

• In Vietnamese, a setting studied under low-resource languages, climate coverage shifts over two decades from incident-driven pollution stories toward policy and energy transition (Nguyen 2026).
• Swiss media coverage of AI and skills diverges from the scientific literature on the same subject (Reveilhac et al. 2026).
• Brazilian discussion of antimicrobial resistance differs across tweets, news and government texts (Susin & Berton 2026).
• In Telegram groups, conspiratorial talk turns up inside everyday themes rather than in isolated pockets (Shimgekar et al. 2026).

Topic clusters also serve as a sampling frame for annotation budgets, as in bias labelling of war news (Jafari et al. 2024). Author profiles built from topics support stylometry (Seroussi et al. 2014).

People and institutions

The canonical probabilistic topic model, LDA, came from David M. Blei, Andrew Y. Ng and Michael I. Jordan (Blei et al. 2003). Jordan is now at the University of California, Berkeley and Inria.

Blei, now at Columbia University, is the most constant figure in the field's lineage:

• dynamic topic models with John D. Lafferty, now at Yale University (Blei & Lafferty 2006);
• the human interpretability study with Jonathan Chang, Jordan Boyd-Graber and others (Chang et al. 2009);
• the embedded topic model with Adji B. Dieng and Francisco J. R. Ruiz, now at Google DeepMind (Dieng et al. 2020).

Thomas Hofmann, now at ETH Zurich, introduced pLSA (Hofmann 1999). Three researchers contributed polylingual topic models (Mimno et al. 2009) and corpus-internal coherence (Mimno et al. 2011):

• David Mimno, now at Cornell University;
• Hanna Wallach, at Microsoft Research New York City;
• Andrew McCallum, at the University of Massachusetts Amherst.

The automatic coherence line owes much to David Newman, Jey Han Lau and Timothy Baldwin (Newman et al. 2010; Lau et al. 2014). Lau and Baldwin are now at The University of Melbourne, and Baldwin also at Mohamed bin Zayed University of Artificial Intelligence.

Contextualised topic models come from Federico Bianchi (now at Stanford University), Silvia Terragni and Dirk Hovy (now at Bocconi University) (Bianchi et al. 2021; Bianchi et al. 2021). Samuel Brody, now at Google, applied LDA-style models to sense induction and to review aspects (Brody & Lapata 2009; Brody & Elhadad 2010). Maarten Grootendorst created BERTopic (Grootendorst 2022).

Open questions

Evaluation remains unsettled, and the questions most often raised are about measurement:

• Likelihood and interpretability can pull in opposite directions (Chang et al. 2009).
• Coherence measures validated on news and Wikipedia may misjudge topics in two kinds of setting. One is specialised scholarly domains (Prouteau et al. 2026). The other is languages such as Vietnamese, where co-occurrence scores may undervalue topic quality (Nguyen 2026).
• Coherence measures fit poorly with span-level topics (Yoon et al. 2026).
• Gains in coherence do not always translate into fewer bad topics. The generalized Pólya urn model raised average coherence, yet its share of bad topics was 13.5% against 16.5% for LDA, a gap that was not statistically significant (Mimno et al. 2011).
• When models output labelled descriptions or activation features instead of word lists, word-based metrics no longer apply. This motivates LLM-judged comparisons (Zheng et al. 2026), which raise their own questions of reliability. These issues connect to wider debates on evaluation and benchmarks.

Model assumptions are also contested:

• Bag-of-words models discard word order and syntax (Bianchi et al. 2021).
• Clustering pipelines often impose one topic per document (Grootendorst 2022), whereas real reviews and survey answers mix several themes within a few sentences (Yoon et al. 2026).
• Choosing K is still largely heuristic, and incremental alternatives are sensitive to the order in which documents arrive (Singaravadivelan et al. 2026).
• Temporal models must balance smooth evolution against stagnant topics. They must also decide whether outliers are noise or nascent themes (Nguyen et al. 2026; Zve et al. 2026).

Language coverage is uneven:

• Cross-lingual models depend on scarce bilingual resources (Xuan et al. 2026).
• Zero-shot transfer still trails a translation baseline at higher topic counts (Bianchi et al. 2021).
• Morphologically rich languages need careful preprocessing (Aftar et al. 2024).
• Topic classification on classical Arabic remains hard even where named entity recognition succeeds (Ahmed et al. 2022).

These issues are a central concern of multilingual NLP.

Finally, the arrival of LLMs reopens the question of what topic modeling is for. Prompt-based systems produce readable, editable topics, but they have three costs. They are expensive to run, they truncate long documents, and they depend on closed models, since open-weight models struggled at the generation stage (Pham et al. 2024). Classical models stay cheap, private and transparent (Xu et al. 2026). In some settings they remain competitive with fine-tuned classifiers and prompted LLMs while keeping higher precision (Madureira et al. 2026). Deciding when a lightweight probabilistic model is enough, and when a whole LLM-driven system is warranted, is likely to define the next phase of the field.

The library in figures