🧑‍🔬 Tom B. Brown

Tom B. Brown works on large neural language models and, more broadly, on how autoregressive generative models change as they grow. The papers here, all from 2020, ask what happens when Transformer models are made larger, trained on more data, and given more compute. They answer through extensive series of training runs, judged by loss on held-out data and by evaluation across many benchmark tasks. The work sits within language models & pretraining.

The most cited is Language Models are Few-Shot Learners, which introduced GPT-3, a very large autoregressive transformer trained on filtered web text. It was tested on a wide range of tasks with no fine-tuning: an instruction and a handful of examples are placed in the prompt, a behaviour the authors call in-context learning. Scaling Laws for Neural Language Models trained families of Transformers on WebText2 while varying model size, dataset size, training time, batch size and architecture shape. It found that loss falls along simple power laws in parameters, data and compute, as long as neither of the other two factors holds it back.

The newest work carries the scaling question beyond text. Scaling Laws for Autoregressive Generative Modeling fits the same kind of curves to images, video, paired captions and images, and procedurally generated mathematics problems. In each domain the loss follows a smooth power law plus a constant, and the authors read that constant as an estimate of the inherent entropy of the data. Brown's papers here come from OpenAI research teams. Frequent co-authors include Jared Kaplan, Tom Henighan, Benjamin Mann, Prafulla Dhariwal and Scott Gray.

Tom B. Brown has 3 papers in our library, in 2020, most of them on Language models & pretraining; often with Jared Kaplan, Benjamin Mann, Prafulla Dhariwal.