🧑‍🔬 Andrej Risteski

Andrej Risteski studies why word embeddings work. The approach treats lexical semantics with probabilistic generative models and proofs rather than with purely empirical comparison. The papers here ask what low-dimensional word vectors, such as those produced by word2vec and GloVe, actually capture. They also ask how such vectors arise from co-occurrence statistics, and how a single vector can stand for a word that has more than one meaning. The work belongs to lexical semantics and word embeddings and also bears on word sense disambiguation.

The most cited is A Latent Variable Model Approach to PMI-based Word Embeddings (2016). It pictures text as generated by a hidden discourse vector that drifts slowly over time. Each word's chance of appearing grows exponentially with how closely its own vector aligns with that hidden state. This account explains why compact vectors reproduce PMI matrices so well through their inner products. It also explains why plain addition and subtraction of vectors are enough to answer analogy questions. The follow-up, Linear Algebraic Structure of Word Senses (2018), carries the same line of reasoning to polysemy. It argues that the embedding of an ambiguous word is a weighted, nonnegative combination of vectors for its separate senses. It supports the claim with a Gaussian variant of the random-walk model.

The newest work here moves from explaining embeddings in general toward the internal structure of word meaning. It shows that a word's vector is linearly related to the averaged embeddings of the contexts in which it occurs.

Risteski is currently at Carnegie Mellon University, after earlier papers written at Princeton University. Collaborators on these papers include Sanjeev Arora, Tengyu Ma, Yingyu Liang and Yuanzhi Li.

Andrej Risteski has 2 papers in our library, in 2016–2018, most of them on Word sense disambiguation, Lexical semantics & word embeddings; often with Sanjeev Arora, Tengyu Ma, Yingyu Liang.