🧑‍🔬 Zeyuan Allen-Zhu

Zeyuan Allen-Zhu works on large language models. The papers here take them from two sides: how to adapt a pretrained model cheaply to new tasks, and what such a model actually learns from its training text. The methods range from changes to how Transformer weights are updated during fine-tuning to controlled experiments in which small models are trained from scratch on synthetic text, so that the influence of the training data can be isolated. Both papers belong to Language models & pretraining, and one also to Efficient models & compression.

The most cited work is LoRA: Low-Rank Adaptation of Large Language Models (2021). Full fine-tuning updates every weight, which for a model the size of GPT-3 means keeping a separate full copy for each task. LoRA instead leaves the pretrained weights frozen. For selected Transformer layers it learns only the change to the weights, written as the product of two narrow low-rank matrices. This sidesteps the extra inference cost of adapter layers and the optimisation difficulties of prefix tuning.

The more recent Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (2023) asks whether a model genuinely extracts facts from its training text or merely memorises answers to familiar questions. To separate the two, GPT-2-style models are trained from scratch on a synthetic collection of biographies of fictional people. They are then fine-tuned on questions about some individuals and tested on the rest. The models can reproduce the biographies word for word. Without varied rephrasings of the same facts in the training text, however, they can hardly retrieve those facts for people held out from fine-tuning.

Allen-Zhu's most frequent co-author in these papers is Yuanzhi Li. Other collaborators include Weizhu Chen, Yelong Shen and Lu Wang.

Zeyuan Allen-Zhu has 2 papers in our library, in 2021–2023, most of them on Language models & pretraining, Efficient models & compression; often with Yuanzhi Li, Lu Wang, Weizhu Chen.