Parameter-efficient fine-tuning of large-scale pre-trained language models

Written for the Research library (claude-opus-5-5) from the paper text. Original — nature.com/articles/s42256-023-00626-4.pdf

Summary

Updating every weight of ever-larger pre-trained language models (PLMs) is becoming too costly in compute and storage, because each downstream task needs its own full model copy. The authors group the many lightweight alternatives (Adapter tuning, BitFit, LoRA, prefix-tuning, prompt-tuning and others) under one name, "delta-tuning": a small set of parameters, either existing or newly inserted, is trained while the frozen backbone stays untouched. They formalise the problem and sort methods into addition-based, specification-based and reparameterization-based families. They offer two theoretical lenses. One treats delta-tuning as optimisation inside a low-dimensional subspace. The other treats the delta parameters as optimal controllers that steer hidden-state propagation from layer to layer. Experiments cover more than 100 NLP tasks drawn from CrossFit and GLUE. Delta-tuning comes close to full fine-tuning, no single method wins everywhere, and mixing methods usually helps. With large enough backbones, even randomly chosen tunable parameters become competitive. The work lowers the entry cost of billion-parameter models and clarifies why few-parameter adaptation succeeds.

Key points

• A manual survey of 1,200 papers from ACL, EMNLP and NAACL (2020–2022) found that only about 0.5–4% actually ran PLMs with over 1 billion parameters in their experiments.
• Fully fine-tuning GPT-3 would touch roughly 175,255 million parameters. LoRA-style low-rank matrices reduce the trainable portion to about 37.7 million.
• Delta-tuning usually converges more slowly than full fine-tuning. Convergence depends more on the delta structure than on the count of tunable parameters, and it improves as model size grows.
• Intrinsic prompt-tuning tunes just 250 free parameters in a shared subspace learned from 100 tasks. It recovers 97% of full prompt-tuning performance on seen tasks and 83% on 20 unseen tasks.
• Code is released as the OpenDelta toolkit (GitHub thunlp/OpenDelta, also archived on Zenodo). Datasets come from INK-USC/CrossFit and Hugging Face GLUE.