📄 Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora

Summary

Many text collections, from newswire to social bookmarking sites like del.icio.us, give each document several human tags, but a tag rarely fits every part of a document equally well. The authors call this the credit attribution problem: deciding which words belong to which tag. Labeled LDA (L-LDA) addresses it by tying LDA topics one-to-one to the observed labels. Each document's topic mixture may only use topics matching its own tags. Training uses collapsed Gibbs sampling, with topic choices for each word limited to the document's label set. For singly labeled documents, the model reduces to Multinomial Naive Bayes. When a document has several labels, each word's count is split among those labels rather than duplicated. On tagged del.icio.us pages, L-LDA produces more interpretable visualizations than plain LDA. It also beats SVMs by more than 3 to 1 at extracting tag-specific snippets. As a multi-label classifier, it stays competitive with one-vs-rest SVMs. This matters because it turns readily available tags into word-level, label-aligned topics.

Key points

• The number of topics equals K, the number of unique labels. A per-document projection matrix L^(d) reduces the Dirichlet prior α to the document's own labels.
• Prior supervised variants do not fit multi-label data in the same way:
  • Supervised LDA and DiscLDA allow only one label per document.
  • MM-LDA allows several labels, but its topics do not map directly onto them.
• When extracting tag-specific snippets from del.icio.us pages, L-LDA's margin of advantage over SVMs exceeds 3 to 1.
• The classification experiment used 4,000 del.icio.us documents with a 20-tag subset:
  • The split was 80/20, averaged over 10 random permutations.
  • The SVM baseline used C = 10.0 and raw term frequency, chosen by 4-fold cross-validation on 3,000 separate documents.
• MacroF1 was about equal for L-LDA and SVM, while L-LDA was clearly ahead on MicroF1. The improvement was significant under a 2-tailed paired t-test at 95%.

Interesting ideas

• I1. Supervision as a constraint. Instead of adding a new output variable, the model simply limits which topics a document may use. This is a cheap way to inject metadata into any mixture model, such as region tags or product categories.
• I2. Splitting credit instead of duplicating it. Unlike one-vs-rest Naive Bayes, each word instance is shared among the document's labels rather than counted fully for every label. This yields cleaner per-label word distributions in multi-tag data.
• I3. Document context resolves word ambiguity. In the design vs. programming example, the inferred document-level mixture θ overrides word likelihoods that favor the other tag. This gives a form of contextual sense disambiguation that could help in tagging or routing tasks.
• I4. Evaluating through a downstream task. The paper judges word-to-tag assignment by tag-specific snippet extraction, not only by classification accuracy. This is a practical evaluation pattern for any model that claims fine-grained attribution.
• I5. Clear routes to extension. Because L-LDA belongs to the LDA family, the authors suggest two extensions: modeling label correlations by combining it with the Correlated Topic Model or Pachinko Allocation, and a semi-supervised setting where only some documents carry labels. The semi-supervised case is useful when annotation is sparse.