Elamur Research
☰
home / papers / 2020 / Topic Modeling in Embedding Spaces

Topic Modeling in Embedding Spaces

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.tacl-1.29.pdf

Summary

Classic topic models such as LDA perform poorly when the vocabulary is large and heavy-tailed. Practitioners usually prune very frequent and very rare terms, which can discard useful words. Dieng, Ruiz and Blei address this with the embedded topic model (ETM), which combines LDA-style document modeling with word embeddings. Words and topics share one low-dimensional embedding space. Each topic's distribution over the vocabulary comes from a softmax of the inner products between the topic vector and each word vector. Embeddings can be supplied pre-trained (e.g., skip-gram) or learned together with the topics. Fitting uses amortized variational inference, which lets the model scale to big corpora. On 20NewsGroup and The New York Times, the ETM was compared with LDA, NVDM, ProdLDA and a ProdLDA variant using pre-fitted embeddings. Among models with interpretable topics, it achieved the best document-completion likelihood and stayed stable as the vocabulary grew. It also produced readable topics when stop words were kept, gathering them into a few dedicated "stop topics." This matters for practical topic modeling with minimal preprocessing.

Key points

  • The ETM represents every topic as a vector located in the same embedding space as the words. A word's probability under a topic is proportional to the exponentiated topic–word inner product.
  • The authors introduce topic diversity, defined as the share of unique words among the top 25 words of all topics. Overall topic quality is diversity multiplied by NPMI-based topic coherence.
  • On 20NewsGroup (11.2K articles, 100 topics), LDA degrades as the vocabulary grows, while the ETM keeps its predictive and topic quality.
  • On New York Times data with stop words retained (vocabulary of 10,283, K = 300), ETM-PWE reached a topic quality of 0.0396. The baselines scored as follows:
    • LDA: 0.0182
    • Δ-NVDM: 0.0187
    • ProdLDA-PWE: 0.0159
  • Code is released at github.com/adjidieng/ETM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Computing & GPUs

Cited by

133 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 132 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00325

@article{dieng2020topic,
  title = {{Topic Modeling in Embedding Spaces}},
  author = {Dieng, Adji B. and Ruiz, Francisco J. R. and Blei, David M.},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.1162/tacl\_a\_00325},
  url = {https://doi.org/10.1162/tacl_a_00325}
}

References

5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John Aitchison and Shir Ming Shen. 1980. Logistic normal distributions: Some properties and uses. Biometrika, 67(2):261–272. find
  2. Kayhan Batmanghelich, Ardavan Saeedi, Karthik Narasimhan, and Sam Gershman. 2016. Non-parametric spherical topic modeling with word embeddings. In Association for Computational Linguistics, volume 2016, page 537. find
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. *Figure 6: A topic containing stop words found by the ETM -PWE on The New York Times. The ETM is robust even in the presence of stop… find
  4. David M. Blei. 2012. Probabilistic topic models. Communications of the ACM, 55(4):77–84. find
  5. David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. 2017. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859–877. \#\# 7 Conclusion find