Elamur Research⌕
☰
home / papers / 2025 / Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer

Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer

Paper · 2025 · Guodong Du, Zitao Fang, Jing Li et al.

Summary

Fine-tuned checkpoints of foundation models often generalize poorly beyond their target domain and contain many redundant weight changes. Earlier work showed that pruning these changes and adding the remainder back onto the pre-trained weights reduces forgetting, eases multi-task merging, and cuts storage. The quality of that pruning step therefore matters.

The authors propose Neural Parameter Search (NPS). It first computes the task vector, meaning the fine-tuned weights minus the pre-trained ones. It then groups the task vector's entries into a fixed number of subspaces ranked by magnitude. A gradient-free evolutionary algorithm assigns each subspace a scaling weight, guided by scores on a small calibration set. Finally, the rescaled vector is magnitude-pruned to a chosen sparsity and added to the base model.

NPS is evaluated in three settings:

  • Knowledge transfer: interpolating the pruned model with the base model (LLaVa).
  • Model merging: combining several pruned models (T5, RoBERTa, CLIP ViT, LLaMa2, PEFT adapters).
  • Checkpoint compression.

Across vision, NLP and multi-modal benchmarks it outperforms prior baselines. This makes it a cheap, practical tool for slimming and reusing fine-tuned models.

Key points

  • Search procedure: the evolutionary search avoids gradient computation entirely, which keeps memory use low. In a ViT-B/32 SVHN example (sparsity 0.1, five subspaces), the subspace weights and accuracy stabilize over generations.
  • Merging results:
    • +4.3% with T5-base.
    • +3.0% (ViT-B/32) and +1.4% (ViT-L/14) on eight CLIP image-classification tasks.
    • +1.3% (Roberta-base) and +0.9% (T5-base) on five emotion-classification domains.
  • LLaMa2 merging: about 0.9% average gain on CMMLU (Chinese), GSM8K (math) and HumanEval (code).
  • Compression: with eight fine-tuned ViT-B/32 models, NPS keeps roughly 99% normalized accuracy while storage grows only gradually with the number of tasks. The paper reports about 40% better compression efficiency on vision tasks than alternatives such as TALL-Mask + TIES.
  • Code: publicly available at github.com/duguodong7/NPS-Pruning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations

Also on the byline: Xiamen University Malaysia.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Guodong Du, Zitao Fang, Jing Li, Junlin Li, Runhua Jiang, Shuyang Yu, Yifei Guo, Yangneng Chen, Sim Kuan Goh, Ho-Kin Tang, Daojing He, Honghai Liu and Min Zhang. 2025. Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1570

@inproceedings{du2025neural,
  title = {{Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer}},
  author = {Guodong Du, Zitao Fang, Jing Li, Junlin Li, Runhua Jiang, Shuyang Yu, Yifei Guo, Yangneng Chen, Sim Kuan Goh, Ho-Kin Tang, Daojing He, Honghai Liu and Min Zhang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1570},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1570}
}

References

5 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. This work was supported by National Science Foundation of China (62476070), Shenzhen Science and Technology Program (JCYJ20241202123503005, GXWD20231128103232001, ZDSYS2023062609 1203008, KQTD2024072910215406) and Department of Science and Technology of Guangdong (2024A1515011540). find
  2. Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: machine learning for text-based emotion prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 579–586. \#\# Limitations Saleh Ashkboos, Maximilian L. Croci, Marcelo…
  3. While our method offers advantages such as gradient-free operation, ease of implementation, and broad applicability, it also has certain limitations: Jinhe Bi, Yifan Wang, Danqi Yan, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2025a. Prism: Self-pruning intrinsic selection method for trainin… — [arXiv:2502.12119](https://arxiv.org/abs/2502.12119) find
  4. Dependence on Pretrained Models: Our approach relies on pretrained models as a reference. If the fine-tuned model deviates significantly from the original, it may hinder effective knowledge transfer, fusion, and compression. find
  5. Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2024. Visual instruction tuning with 500x fewer parameters through modality linear representation-steering. arXiv preprint arXiv:2412.12359. — [arXiv:2412.12359](https://arxiv.org/abs/2412.12359) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/du-2025-neural-parameter-search-slimmer-fine/