🧑‍🔬 Benjamin Mann

Benjamin Mann's papers here concern large autoregressive models and how their behaviour changes with scale. Both works ask what happens when transformer models grow in size and training compute. One question is whether a bigger language model can take on new tasks from a handful of examples. The other is whether the regular gains seen in language also appear for other kinds of data. The methods are empirical: training families of models of different sizes and measuring their performance or loss across many benchmarks and domains. The work sits within the topic of Language models & pretraining.

The most cited of these papers is Language Models are Few-Shot Learners (2020), which introduced GPT-3. The paper trained a very large transformer on filtered web text and tested it on a broad range of datasets without updating its weights. The tasks and examples were supplied only through the prompt, an approach the authors named in-context learning. The second paper, Scaling Laws for Autoregressive Generative Modeling (2020), carried the scaling analysis beyond text. It trained decoder-only transformers on images, video, paired captions and images, and generated mathematics problems. In every domain, loss declined smoothly as models and budgets grew. The decline fit a power-law curve that levels off at a fixed floor, which the authors took as an estimate of the data's own entropy.

Both papers date from 2020 and come from research teams at OpenAI. They form a single line of inquiry into scale as a route to more general capability. Mann's co-authors on these papers include Tom B. Brown, Jared Kaplan and Prafulla Dhariwal, each of whom appears on both works. Readers new to this research may want to start with the GPT-3 paper and then turn to the multi-domain scaling study.

Benjamin Mann has 2 papers in our library, in 2020, most of them on Language models & pretraining; often with Jared Kaplan, Prafulla Dhariwal, Tom B. Brown.