🧑‍🔬 Sanjeev Arora

Sanjeev Arora's papers here study the mathematics of how machine learning models represent language. The earlier work builds generative, latent-variable models of text. These models explain why word vectors learned from co-occurrence statistics behave as they do, and the papers back the explanations with proofs rather than experiments alone. The themes fall under lexical semantics and word embeddings and word sense disambiguation. The more recent work asks how the choice of training data shapes large language models.

The most cited of these papers is A Latent Variable Model Approach to PMI-based Word Embeddings (2016). It treats text as produced by a hidden discourse vector that drifts slowly as a random walk, with each word emitted according to its inner product with that vector. The model accounts for two observations: low-dimensional vectors closely approximate PMI matrices, and analogies can be solved with vector arithmetic. Linear Algebraic Structure of Word Senses, with Applications to Polysemy (2018) carries this framework over to ambiguous words. It argues that the embedding of a word with several meanings is a weighted, nonnegative combination of vectors for its separate senses. Using a Gaussian version of the walk model, it proves that a word's vector is a linear transformation of the averaged embeddings of its contexts.

The newest work, LESS: Selecting Influential Data for Targeted Instruction Tuning (2024), moves into language models and pretraining. It addresses a setting where a model needs only a narrow set of skills, and where training on a broad instruction mixture can even weaken those skills. Given a few examples of the desired capability, the method ranks items in a large instruction pool by a gradient-based influence estimate. That estimate is adapted to the Adam optimiser and compared by cosine similarity.

Arora is based at Princeton University. Earlier bylines include the Institute for Advanced Study and the University of California, Berkeley. Co-authors on these papers include Andrej Risteski, Tengyu Ma, Yingyu Liang and Yuanzhi Li, as well as Mengzhou Xia and Danqi Chen.

Sanjeev Arora has 3 papers in our library, in 2016–2024, most of them on Word sense disambiguation, Language models & pretraining, Lexical semantics & word embeddings; often with Andrej Risteski, Tengyu Ma, Yingyu Liang.