🧑‍🔬 Charles Sutton

Charles Sutton works on large language models and what they can do with both natural language and source code. Sutton's papers here ask how a model's size shapes its ability to handle new tasks from only a few examples, and how far general-purpose models can go at writing working programs in a mainstream language such as Python. The approach is empirical: training or testing Transformer models across a range of sizes and measuring them on broad sets of benchmarks. The work falls under Language models & pretraining and Code & programming languages.

The most cited paper here is PaLM: Scaling Language Modeling with Pathways. It describes training a very large dense decoder-only Transformer, spread across TPU v4 pods with the Pathways system. The resulting model gave strong few-shot results in language understanding and generation, translation, multilingual tasks and code. The next is Program Synthesis with Large Language Models. It introduced two benchmarks, MBPP and MathQA-Python, and showed that accuracy at writing short Python programs rises steadily with model scale, under both few-shot prompting and fine-tuning.

These two papers, from 2021 and 2022, also mark the most recent direction of the work. Both move away from earlier program synthesis built around narrow, purpose-made languages, toward general models evaluated on everyday programming and reasoning problems. Sutton is currently at Google. Earlier affiliations include the University of Edinburgh, the University of Massachusetts Amherst, the University of California, Berkeley, and The Alan Turing Institute. Co-authors on the papers here include Maarten Bosma, Jacob Austin and Aakanksha Chowdhery, along with others such as Quoc V. Le and Henryk Michalewski.

Charles Sutton has 2 papers in our library, in 2021–2022, most of them on Language models & pretraining, Code & programming languages; often with Maarten Bosma, Aakanksha Chowdhery, Adam Roberts.