🧑‍🔬 Michael Matena

Michael Matena works on transfer learning for natural language processing and on ways of combining neural networks that have already been trained. Their papers here run from large controlled comparisons of how language models are pre-trained and adapted to a probabilistic treatment of parameter averaging. Both lines ask how knowledge held in one trained model can be reused for, or joined with, another.

The most cited of these papers is Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019). It recasts translation, classification, question answering and summarization alike as mapping one text to another, so that one model, loss and set of hyperparameters can serve every task. The authors used this setting not to propose a new method but to compare existing choices carefully: pre-training objectives, Transformer variants, unlabeled corpora, fine-tuning schemes and multi-task training. They also built the C4 corpus, a cleaned web crawl, for the experiments. The work was a joint effort with Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee and others.

The later paper, Merging Models with Fisher-Weighted Averaging (2021), written with Raffel, looks again at averaging the weights of models that share an architecture and initialization. It treats merging as a search for parameters that approximately maximize the joint likelihood under each model's posterior. In this view, simple averaging corresponds to assuming an isotropic Gaussian. A Laplace approximation with diagonal Fisher information replaces that assumption, so each parameter is averaged with weights given by its Fisher values. Matena's current institution is the University of North Carolina at Chapel Hill, where their bylines appear from 2021.

Michael Matena has 2 papers in our library, in 2019–2021; often with Colin Raffel, Adam Roberts, Katherine Lee.