🧑‍🔬 Yanqi Zhou

Yanqi Zhou works on large neural networks for language and on how such systems behave as they grow. Their papers here return to the same practical questions: how far performance improves with more data and larger models, how to compare training choices fairly, and how to add model capacity without a matching rise in compute. The methods are mostly large empirical studies and new model designs, centred on Language models & pretraining.

The most cited of these papers, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019), treats every language task as turning one text into another. That framing lets a single model and training setup serve tasks from translation to summarization. On this basis the paper systematically compares pre-training objectives, architectures, corpora and fine-tuning strategies, and it introduces the C4 web corpus. An earlier study, Deep Learning Scaling is Predictable, Empirically (2017), trained models for speech, vision, language modeling and translation on growing slices of data. It found that accuracy gains follow power-law curves that can be measured and forecast, and that these curves do not match the exponents classical theory predicts.

The newest work here, GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021), turns to the cost of very large decoder-only models. It replaces some feed-forward layers with sparsely gated expert layers, so that each token activates only a small part of the network. The models are trained on a large, quality-filtered text collection.

Zhou is currently at Google. Earlier bylines list Baidu, where the scaling study was done, and Princeton University. Co-authors on these papers include Colin Raffel and Sharan Narang, as well as Joel Hestness and Dmitry Lepikhin.

Yanqi Zhou has 3 papers in our library, in 2017–2021, most of them on Language models & pretraining; often with Sharan Narang, Adam Roberts, Adams Wei Yu.