🧑‍🔬 Eric Hambro

Eric Hambro's papers here deal with language models and pretraining. They cover two connected questions: how to build foundation models that perform well while staying cheap to run, and how the later stages of fine-tuning change what those models do. The methods are empirical. The work trains models on openly available text, and it compares the stages of a training pipeline under controlled conditions to see what each one adds.

The most cited work is LLaMA: Open and Efficient Foundation Language Models (2023). It argues that inference cost, rather than training budget alone, should guide model design once a model is deployed widely. The paper trains a family of transformer models on many more tokens than the Chinchilla scaling laws suggest. The training corpus draws only on public sources such as filtered CommonCrawl, Wikipedia, GitHub, arXiv and books.

The other paper from the same year, Understanding the Effects of RLHF on LLM Generalisation and Diversity, turns from pretraining to alignment. It takes apart the usual RLHF pipeline of supervised fine-tuning, reward modelling and PPO-based reinforcement learning. It then measures how each stage affects robustness to unfamiliar inputs and the variety of generated outputs, using best-of-N sampling to separate the reward model's influence from that of the optimiser.

Hambro's bylines between 2021 and 2023 list University College London. The LLaMA work was written with a Meta AI team including Hugo Touvron, Gautier Izacard and Armand Joulin. The RLHF study was co-authored with Roberta Raileanu and Edward Grefenstette, among others.

Eric Hambro has 2 papers in our library, in 2023, most of them on Language models & pretraining; often with Armand Joulin, Aurelien Rodriguez, Baptiste Rozière.