🧑‍🔬 Sam McCandlish

Sam McCandlish's papers here deal with language models and pretraining, with a recurring interest in how model behaviour depends on scale. The typical method is empirical. Families of Transformer models are trained across wide ranges of size and data, and simple quantitative relationships are fitted to the results. A later strand moves from training loss to what models actually say, comparing their answers with human survey responses.

The most cited work is Scaling Laws for Neural Language Models (2020). It trained a long series of Transformers on the WebText2 corpus while varying model size, dataset size, training length, batch size and network shape. It found that test loss falls as a power law in parameters, data and compute, each considered on its own, as long as neither of the other two becomes the limiting factor. A follow-up, Scaling Laws for Transfer (2021), asks what pre-training is worth when the downstream task has little data. It expresses that benefit as an equivalent quantity of fine-tuning data and studies it across several orders of magnitude in model and dataset size.

The newest paper, Towards Measuring the Representation of Subjective Global Opinions in Language Models (2023), turns to the question of whose views a model's answers on contested social topics resemble. It introduces GlobalOpinionQA, a multiple-choice set built from the Pew Global Attitudes survey and the World Values Survey. The paper then compares a dialogue model's answer probabilities with country-level human answer distributions under several prompting conditions. The model studied was trained with RLHF and Constitutional AI.

McCandlish's current affiliations are the Machine Intelligence Research Institute in Berkeley and the University of Oxford. Papers from 2021 and 2022 list Anthropic. Frequent co-authors on the scaling work include Jared Kaplan, Tom Henighan and Danny Hernandez.

Sam McCandlish has 3 papers in our library, in 2020–2023, most of them on Language models & pretraining; often with Danny Hernandez, Jared Kaplan, Tom Henighan.