🧑‍🔬 Paul Barham

Paul Barham's papers here deal with the systems that make large-scale machine learning possible. They focus on how computation can be expressed, distributed and run efficiently across many devices, and on what becomes possible when models are trained at very large scale. The work moves between system design and its use in language models and pretraining.

The most cited of these papers is the white paper on TensorFlow (2016). It describes the successor to Google's earlier DistBelief training system, which had proved inflexible for research and hard to move onto small devices. TensorFlow serves both as an interface for writing machine learning computations and as a runtime for executing them. Programs are stateful dataflow graphs whose nodes are operations with kernels for CPUs or GPUs and whose edges carry typed tensors.

The newer direction is shown by PaLM: Scaling Language Modeling with Pathways (2022). This work trained a very large dense decoder-only Transformer to study how model scale affects few-shot performance without task-specific fine-tuning. Training was spread over TPU v4 pods using the Pathways system and reached high hardware utilization without pipeline parallelism. The resulting model gave strong few-shot results across language, code, translation and multilingual benchmarks. Co-authors on these papers include Aakanksha Chowdhery, Jeff Dean, Sharan Narang and Ian Goodfellow.

Barham's bylines run from the University of Cambridge (1995–2002) to Microsoft Research in Cambridge and Microsoft, and most recently to Google (2015–2022).

Paul Barham has 2 papers in our library, in 2016–2022, most of them on Language models & pretraining; often with Aakanksha Chowdhery, Adam Roberts, Charles Sutton.