🧑‍🔬 Jeffrey Pennington

Jeffrey Pennington works on learning vector representations of language. The papers here span 2011–2014 and return to one question: how continuous vectors for words and phrases can capture meaning, and how such vectors can be learned from text. One approach draws on co-occurrence statistics gathered across a whole corpus. Another composes word vectors step by step into representations of longer spans. The work also touches on Sentiment & emotion, using learned sentence representations to predict the feelings a text expresses.

The most cited paper is GloVe: Global Vectors for Word Representation (2014). It asks why word vectors tend to encode semantic relationships as linear offsets. Its answer is that ratios between co-occurrence probabilities, rather than the raw probabilities, are what carry meaning. On that basis it builds a weighted least-squares model trained on global counts. The model joins the strengths of matrix-factorization methods such as LSA with those of window-based methods such as skip-gram. Earlier, Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions (2011) learned sentence vectors by repeatedly merging neighbouring word vectors into a tree, with no parser or hand-built sentiment lexicon. It then predicted distributions over sentiment from those vectors.

The newest paper here, GloVe, marks a shift from a compositional model built for one task toward general-purpose word vectors learned from corpus-wide statistics. Both papers date from Pennington's time at Stanford University. Richard Socher and Christopher D. Manning appear as co-authors on each, and Andrew Y. Ng and Eric Huang also contributed. Later bylines, from 2015 to 2022, list Google.

Jeffrey Pennington has 2 papers in our library, in 2011–2014, most of them on Sentiment & emotion; often with Christopher D. Manning, Richard Socher, Andrew Y. Ng.