Latent Dirichlet Allocation

Summary

The paper addresses how to obtain compact, statistically meaningful representations of documents and other collections of discrete data. Earlier approaches fall short in different ways. tf-idf and LSI compress data but reveal little probabilistic structure. Hofmann's pLSI has no generative model at the document level, so its parameter count grows with the corpus, it overfits, and it cannot naturally score unseen documents. Starting from de Finetti's theorem on exchangeable variables, the authors propose latent Dirichlet allocation (LDA), a three-level hierarchical Bayesian model. In LDA, each document draws a topic mixture from a Dirichlet prior, and every word is produced by first sampling a topic and then a word from that topic's distribution. Exact inference is intractable, so they derive a variational approximation and an EM procedure for empirical Bayes parameter estimation. Experiments cover document modeling, text classification and collaborative filtering, with mixture of unigrams and pLSI as baselines. In the collaborative filtering experiment, LDA gives the best predictive perplexity, even though both baselines receive overfitting corrections. Because LDA is modular and extensible, it can serve as a building block for richer probabilistic models.

Key points

Generative process: document length N is drawn from a Poisson distribution, topic proportions θ from Dir(α), and each word comes from a topic-specific multinomial. Word probabilities form a k×V matrix β, with the number of topics k fixed in advance.
Baselines are the mixture of unigrams model and pLSI (the aspect model). A key criticism of pLSI is that its parameters scale linearly with corpus size.
The collaborative filtering setup uses EachMovie, keeping only users with at least 100 positive ratings (4 or more of 5 stars). This yields 3300 training and 390 test users and a vocabulary of 1600 movies, evaluated by predictive perplexity on a held-out movie.
Inference relies on a convexity-based variational method. The authors point to expectation propagation (Minka and Lafferty) and MCMC (Griffiths and Steyvers) as alternative inference approaches.
Proposed extensions include Gaussian or other non-multinomial emissions, mixtures of Dirichlets, time-series topics, and conditioning on paragraph or sentence features. Paired LDA modules have already been used to model images with their captions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

1948 papers of our library cite it (of the 140673 whose bibliographies we read); 50 of them have a page here, 1898 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{blei2003latent,
  title = {{Latent Dirichlet Allocation}},
  author = {Blei, David M. and Ng, Andrew Y. and Jordan, Michael I.},
  booktitle = {JMLR 3},
  year = {2003},
  doi = {10.5555/944919.944937},
  url = {https://doi.org/10.5555/944919.944937}
}

References

13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

URL M. Jordan, Z. Ghahramani, T. Jaakkola, and L. Saul. Introduction to variational methods for graphical models. Machine Learning, 37:183–233, 1999. find
R. Kass and D. Steffey. Approximate Bayesian inference in conditionally independent hierarchical models (parametric empirical Bayes models). Journal of the American Statistical Association, 84 (407):717–726, 1989. find
M. Leisink and H. Kappen. General lower bounds based on computer generated higher order expansions. In Uncertainty in Artificial Intelligence, Proceedings of the Eighteenth Conference, 2002. find
T. Minka. Estimating a Dirichlet distribution. Technical report, M.I.T., 2000. find
T. P. Minka and J. Lafferty. Expectation-propagation for the generative aspect model. In Uncertainty in Artificial Intelligence (UAI), 2002. find
C. Morris. Parametric empirical Bayes inference: Theory and applications. Journal of the American Statistical Association, 78(381):47–65, 1983. With discussion. find
K. Nigam, J. Lafferty, and A. McCallum. Using maximum entropy for text classification. IJCAI-99 Workshop on Machine Learning for Information Filtering, pages 61–67, 1999. find
K. Nigam, A. McCallum, S. Thrun, and T. Mitchell. Text classification from labeled and unlabeled documents using EM. Machine Learning, 39(2/3):103–134, 2000. find
C. Papadimitriou, H. Tamaki, P. Raghavan, and S. Vempala. Latent semantic indexing: A probabilistic analysis. pages 159–168, 1998. find
A. Popescul, L. Ungar, D. Pennock, and S. Lawrence. Probabilistic models for unified collaborative and content-based recommendation in sparse-data environments. In Uncertainty in Artificial Intelligence, Proceedings of the Seventeenth Conference, 2001. find
J. Rennie. Improving multi-class text classification with naive Bayes. Technical Report AITR-2001-004, M.I.T., 2001. find
G. Ronning. Maximum likelihood estimation of Dirichlet distributions. Journal of Statistcal Computation and Simulation, 34(4):215–221, 1989. find
G. Salton and M. McGill, editors. Introduction to Modern Information Retrieval. McGraw-Hill, 1983. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/

📄 Latent Dirichlet Allocation