🧑‍🔬 Chris J. Maddison

Chris J. Maddison works in machine learning, and the papers here span two quite different questions. One is how to train models that contain discrete random choices, where the usual gradient machinery breaks down. The other is how to forecast what large language models will be able to do without paying to train them. Both are approached through probabilistic modelling and careful statistical analysis, whether of distributions on the probability simplex or of benchmark results gathered across many existing systems.

The most cited of Maddison's papers here is The Concrete Distribution from 2016, written with Mnih and Yee Whye Teh. It starts from the fact that the reparameterization trick lets gradients flow through continuous stochastic nodes but not discrete ones, which usually forces a fallback to score-function estimators. Those estimators are unbiased but tend to be noisy. The paper's answer is a family of continuous distributions on the simplex, with densities in closed form, that stand in for categorical variables during optimisation. The more recent Observational Scaling Laws from 2024 turns to language models and pretraining and to evaluation and benchmarks. Rather than training a ladder of models of increasing size, it draws on around a hundred publicly available models and their leaderboard scores. It then uses a principal-components decomposition to recover a small set of general capability measures that scale with the logarithm of compute within each model family.

Maddison is currently at the University of Toronto and the Vector Institute. Earlier bylines include the University of British Columbia, Google DeepMind, Google and the University of Oxford. Co-authors on the papers here include Yee Whye Teh and Tatsunori Hashimoto.

Chris J. Maddison has 2 papers in our library, in 2016–2024, most of them on Language models & pretraining, Evaluation & benchmarks; often with Tatsunori Hashimoto, Yee Whye Teh.