Elamur Research
☰
home / papers / 2024 / Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2403.14608

Summary

Updating every weight of a model with billions of parameters costs more than most hardware can handle. This survey covers parameter-efficient fine-tuning (PEFT), where the pre-trained backbone stays mostly frozen and only a small set of parameters is trained for a new task. The authors first walk through how a decoder-only model computes its output, using LLaMA-7B as the example: attention, FFN, RoPE and the memory cost of the KV-cache. They then group PEFT methods into four families by where they change that computation. Additive methods insert new modules or alter activations, for example adapters and soft prompts. Selective methods retrain a subset of existing weights. Reparameterized methods, such as LoRA, train a low-rank form that is merged back so inference stays fast. Hybrid methods combine these. Later chapters cover ways to cut costs further, uses of PEFT beyond text, and system designs for running it in practice. The survey offers no new benchmark result. Its value is that it is a single reference covering both algorithms and systems, which helps teams adapting large models on limited compute.

Key points

  • The four-way taxonomy is additive, selective, reparameterized and hybrid PEFT. LoRA, adapters on the FFN and soft prompts on decoder inputs serve as the main examples.
  • The case for PEFT rests on scale. Models grew from about 1.5B parameters (GPT-2) to 175B (GPT-3), and full fine-tuning at that size can need thousands of GPUs. KV-cache memory is given as L × 2 × b × l × d_head × n_head.
  • Ways to cut costs further include KV-cache management, pruning, quantization and memory-efficient training.
  • PEFT is applied to LLMs, Vision Transformers, vision-language models and diffusion models. Examples include ControlNet-style control, T2I-Adapter, IP-Adapter, textual inversion and LCM-LoRA.
  • On the systems side it covers centralized serving and distributed and parallel tuning (PetS, Punica, dLoRA, DLoRA, Offsite-tuning), and lists the HuggingFace peft and Adapters libraries. The version summarized is arXiv v7 (Sep 2024).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

141 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.14608 · arXiv 2403.14608

@misc{han2024parameter,
  title = {{Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey}},
  author = {Han, Zeyu and Gao, Chao and Liu, Jinyang and Zhang, Jeff and Zhang, Sai Qian},
  year = {2024},
  doi = {10.48550/arxiv.2403.14608},
  eprint = {2403.14608},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.14608}
}

References

261 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. find
  2. Y. Zhuang, Y. Yu, K. Wang, H. Sun, and C. Zhang, “Toolqa: A dataset for llm question answering with external tools,” arXiv preprint arXiv:2306.13304, 2023. find our card
  3. W. Zhu, H. Liu, Q. Dong, J. Xu, L. Kong, J. Chen, L. Li, and S. Huang, “Multilingual machine translation with large language models: Empirical results and analysis,” arXiv preprint arXiv:2304.04675, 2023. find our card
  4. M. U. Hadi, R. Qureshi, A. Shah, M. Irfan, A. Zafar, M. Shaikh, N. Akhtar, J. Wu, and S. Mirjalili, “A survey on large language models: Applications, challenges, limitations, and practical usage,” TechRxiv, 2023. find
  5. B. Xu, X. Liu, H. Shen, Z. Han, Y. Li, M. Yue, Z. Peng, Y. Liu, Z. Yao, and D. Xu, “Gentopia: A collaborative platform for tool-augmented llms,” arXiv preprint arXiv:2308.04030, 2023. find
  6. G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “Camel: Communicative agents for ”mind” exploration of large language model society,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. find our card
  7. Q. Wu, G. Bansal, J. Zhang, Y. Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang, “Autogen: Enabling next-gen llm applications via multi-agent conversation framework,” arXiv preprint arXiv:2308.08155, 2023. find our card
  8. H. Zhang, X. Liu, and J. Zhang, “Summit: Iterative text summarization via chatgpt,” arXiv preprint arXiv:2305.14835, 2023. find
  9. B. Zhang and R. Sennrich, “Root mean square layer normalization,” Advances in Neural Information Processing Systems, vol. 32, 2019. find
  10. J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” arXiv preprint arXiv:2104.09864, 2021. find
  11. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” arXiv preprint arXiv:1804.07461, 2018. find our card
  12. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” in EMNLP, 2018. find
  13. Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi, “Piqa: Reasoning about physical commonsense in natural language,” in Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020. find our card
  14. M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi, “Socialiqa: Commonsense reasoning about social interactions,” arXiv preprint arXiv:1904.09728, 2019. find our card
  15. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “Hellaswag: Can a machine really finish your sentence?” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. find
  16. C. e. a. Clark, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” in NAACL, 2019. find our card
  17. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi, “Winogrande: An adversarial winograd schema challenge at scale,” Communications of the ACM, vol. 64, no. 9, pp. 99–106, 2021. find
  18. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv:1803.05457v1, 2018. find
  19. W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev et al., “The kinetics human action video dataset,” arXiv preprint arXiv:1705.06950, 2017. — [arXiv:1705.06950](https://arxiv.org/abs/1705.06950) find
  20. R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag et al., “The” something something” video database for learning and evaluating visual common sense,” in Proceedings of the IEEE international conference on computer vi… find
  21. H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, “Hmdb: a large video database for human motion recognition,” in 2011 International conference on computer vision. IEEE, 2011, pp. 2556–2563. find
  22. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 2014, pp. 740–755. find our card
  23. B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 633– 641. find
  24. M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International journal of computer vision, vol. 88, pp. 303–338, 2010. find
  25. N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen et al., “Parameter-efficient fine-tuning of large-scale pre-trained language models,” Nature Machine Intelligence, vol. 5, no. 3, pp. 220–235, 2023. find our card
  26. L. Xu, H. Xie, S.-Z. J. Qin, X. Tao, and F. L. Wang, “Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment,” arXiv preprint arXiv:2312.12148, 2023. — [arXiv:2312.12148](https://arxiv.org/abs/2312.12148) find
  27. G. Pu, A. Jain, J. Yin, and R. Kaplan, “Empirical analysis of the strengths and weaknesses of peft techniques for llms,” arXiv preprint arXiv:2304.14999, 2023. — [arXiv:2304.14999](https://arxiv.org/abs/2304.14999) find
  28. OpenAI, “Sharegpt,” https://sharegpt.com/, 2023. ↗
  29. Microsoft, “Microsoft azure function trace,” https://github.com/Azure/AzurePublicDataset, 2023. ↗
  30. I. S. Moreno, P. Garraghan, P. Townend, and J. Xu, “Analysis, modeling and simulation of workload patterns in a large-scale utility cloud,” IEEE Transactions on Cloud Computing, vol. 2, no. 2, pp. 208–221, 2014. find
  31. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in International Conference on Machine Learning. PMLR, 2019, pp. 2790–2799. find
  32. J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a unified view of parameter-efficient transfer learning,” arXiv preprint arXiv:2110.04366, 2021. find our card
  33. Y. Zhu, J. Feng, C. Zhao, M. Wang, and L. Li, “Counter-interference adapter for multilingual machine translation,” arXiv preprint arXiv:2104.08154, 2021. find
  34. T. Lei, J. Bai, S. Brahma, J. Ainslie, K. Lee, Y. Zhou, N. Du, V. Y. Zhao, Y. Wu, B. Li et al., “Conditional adapters: Parameter-efficient transfer learning with fast inference,” arXiv preprint arXiv:2304.04947, 2023. — [arXiv:2304.04947](https://arxiv.org/abs/2304.04947) find
  35. J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych, “Adapterfusion: Non-destructive task composition for transfer learning,” arXiv preprint arXiv:2005.00247, 2020. find our card
  36. Y. Wang, S. Mukherjee, X. Liu, J. Gao, A. H. Awadallah, and J. Gao, “Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models,” arXiv preprint arXiv:2205.12410, vol. 1, no. 2, p. 4, 2022. — [arXiv:2205.12410](https://arxiv.org/abs/2205.12410) find
  37. H. Zhao, J. Fu, and Z. He, “Prototype-based hyperadapter for sample-efficient multi-task tuning,” arXiv preprint arXiv:2310.11670, 2023. find
  38. A. Chronopoulou, M. E. Peters, A. Fraser, and J. Dodge, “Adaptersoup: Weight averaging to improve generalization of pretrained language models,” arXiv preprint arXiv:2302.07027, 2023. find
  39. S. He, R.-Z. Fan, L. Ding, L. Shen, T. Zhou, and D. Tao, “Mera: Merging pretrained adapters for few-shot learning,” arXiv preprint arXiv:2308.15982, 2023. find
  40. R. K. Mahabadi, S. Ruder, M. Dehghani, and J. Henderson, “Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks,” arXiv preprint arXiv:2106.04489, 2021. find
  41. X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” arXiv preprint arXiv:2101.00190, 2021. find
  42. J. Li, W. Aitken, R. Bhambhoria, and X. Zhu, “Prefix propagation: Parameter-efficient tuning for long sequences,” arXiv preprint arXiv:2305.12086, 2023. find
  43. X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang, “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,” arXiv preprint arXiv:2110.07602, 2021. find our card
  44. Z.-R. Zhang, C. Tan, H. Xu, C. Wang, J. Huang, and S. Huang, “Towards adaptive prefix tuning for parameter-efficient language model fine-tuning,” arXiv preprint arXiv:2305.15212, 2023. find
  45. X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang, “Gpt understands, too,” arXiv preprint arXiv:2103.10385, 2021. find our card
  46. B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” arXiv preprint arXiv:2104.08691, 2021. find
  47. F. Ma, C. Zhang, L. Ren, J. Wang, Q. Wang, W. Wu, X. Quan, and D. Song, “Xprompt: Exploring the extreme of prompt tuning,” arXiv preprint arXiv:2210.04457, 2022. find
  48. Z. Wu, S. Wang, J. Gu, R. Hou, Y. Dong, V. Vydiswaran, and H. Ma, “Idpg: An instance-dependent prompt generation method,” arXiv preprint arXiv:2204.04497, 2022. find
  49. X. Liu, T. Sun, X. Huang, and X. Qiu, “Late prompt tuning: A late prompt could be better than many prompts,” arXiv preprint arXiv:2210.11292, 2022. find
  50. W. Zhu and M. Tan, “Spt: Learning to selectively insert prompts for better prompt tuning,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 11 862– 11 878. find
  51. Q. Wang, Y. Mao, J. Wang, H. Yu, S. Nie, S. Wang, F. Feng, L. Huang, X. Quan, Z. Xu et al., “Aprompt: Attention prompt tuning for efficient adaptation of pre-trained language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 9147–9160. find
  52. T. Vu, B. Lester, N. Constant, R. Al-Rfou, and D. Cer, “Spot: Better frozen model adaptation through soft prompt transfer,” arXiv preprint arXiv:2110.07904, 2021. find our card
  53. Y. Su, X. Wang, Y. Qin, C.-M. Chan, Y. Lin, H. Wang, K. Wen, Z. Liu, P. Li, J. Li et al., “On transferability of prompt tuning for natural language processing,” arXiv preprint arXiv:2111.06719, 2021. find
  54. J. Wu, T. Yu, R. Wang, Z. Song, R. Zhang, H. Zhao, C. Lu, S. Li, and R. Henao, “Infoprompt: Information-theoretic soft prompt tuning for natural language understanding,” arXiv preprint arXiv:2306.04933, 2023. — [arXiv:2306.04933](https://arxiv.org/abs/2306.04933) find
  55. L. Chen, H. Huang, and M. Cheng, “Ptp: Boosting stability and performance of prompt tuning with perturbation-based regularizer,” arXiv preprint arXiv:2305.02423, 2023. find
  56. Y. Qin, X. Wang, Y. Su, Y. Lin, N. Ding, J. Yi, W. Chen, Z. Liu, J. Li, L. Hou et al., “Exploring universal intrinsic task subspace via prompt tuning,” arXiv preprint arXiv:2110.07867, 2021. — [arXiv:2110.07867](https://arxiv.org/abs/2110.07867) find
  57. J.-Y. Choi, J. Kim, J.-H. Park, W.-L. Mok, and S. Lee, “Smop: Towards efficient and effective prompt tuning with sparse mixture-of-prompts,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 14 306–14 316. find
  58. Z. Shi and A. Lipani, “Dept: Decomposed prompt tuning for parameter-efficient fine-tuning,” arXiv preprint arXiv:2309.05173, 2023. — [arXiv:2309.05173](https://arxiv.org/abs/2309.05173) find
  59. H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 1950–1965, 2022. find our card
  60. T. Zadouri, A. Üstün, A. Ahmadian, B. Ermiş, A. Locatelli, and S. Hooker, “Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning,” arXiv preprint arXiv:2309.05444, 2023. find
  61. D. Lian, D. Zhou, J. Feng, and X. Wang, “Scaling & shifting your features: A new baseline for efficient model tuning,” Advances in Neural Information Processing Systems, vol. 35, pp. 109–123, 2022. find
  62. X. Lu, F. Brahman, P. West, J. Jang, K. Chandu, A. Ravichander, L. Qin, P. Ammanabrolu, L. Jiang, S. Ramnath et al., “Inference-time policy adapters (ipa): Tailoring extreme-scale lms without fine-tuning,” arXiv preprint arXiv:2305.15065, 2023. find
  63. D. Guo, A. M. Rush, and Y. Kim, “Parameter-efficient transfer learning with diff pruning,” arXiv preprint arXiv:2012.07463, 2020. find our card
  64. N. Lawton, A. Kumar, G. Thattai, A. Galstyan, and G. V. Steeg, “Neural architecture search for parameter-efficient fine-tuning of large pre-trained language models,” arXiv preprint arXiv:2305.16597, 2023. find
  65. B. Liao, Y. Meng, and C. Monz, “Parameter-efficient fine-tuning without introducing new latency,” arXiv preprint arXiv:2305.16742, 2023. find
  66. Y.-L. Sung, V. Nair, and C. A. Raffel, “Training neural networks with fixed sparse masks,” Advances in Neural Information Processing Systems, vol. 34, pp. 24 193–24 205, 2021. find
  67. S. S. S. Das, R. H. Zhang, P. Shi, W. Yin, and R. Zhang, “Unified low-resource sequence labeling by sample-aware dynamic sparse fine-tuning,” arXiv preprint arXiv:2311.03748, 2023. — [arXiv:2311.03748](https://arxiv.org/abs/2311.03748) find
  68. A. Ansell, E. M. Ponti, A. Korhonen, and I. Vulić, “Composable sparse fine-tuning for cross-lingual transfer,” arXiv preprint arXiv:2110.07560, 2021. find
  69. Z. Fu, H. Yang, A. M.-C. So, W. Lam, L. Bing, and N. Collier, “On the effectiveness of parameter-efficient fine-tuning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 11, 2023, pp. 12 799–12 807. find
  70. R. Xu, F. Luo, Z. Zhang, C. Tan, B. Chang, S. Huang, and F. Huang, “Raise a child in large language model: Towards effective and generalizable fine-tuning,” arXiv preprint arXiv:2109.05687, 2021. find
  71. D. Vucetic, M. Tayaranian, M. Ziaeefard, J. J. Clark, B. H. Meyer, and W. J. Gross, “Efficient fine-tuning of bert models on the edge,” in 2022 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 2022, pp. 1838–1842. find
  72. E. B. Zaken, S. Ravfogel, and Y. Goldberg, “Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,” arXiv preprint arXiv:2106.10199, 2021. find our card
  73. M. Gheini, X. Ren, and J. May, “Cross-attention is all you need: Adapting pretrained transformers for machine translation,” arXiv preprint arXiv:2104.08771, 2021. find
  74. H. He, J. Cai, J. Zhang, D. Tao, and B. Zhuang, “Sensitivity-aware visual parameter-efficient fine-tuning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 11 825– 11 835. find
  75. A. Aghajanyan, L. Zettlemoyer, and S. Gupta, “Intrinsic dimensionality explains the effectiveness of language model fine-tuning,” arXiv preprint arXiv:2012.13255, 2020. find our card
  76. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021. find
  77. R. Karimi Mahabadi, J. Henderson, and S. Ruder, “Compacter: Efficient low-rank hypercomplex adapter layers,” Advances in Neural Information Processing Systems, vol. 34, pp. 1022–1035, 2021. find
  78. A. Edalati, M. Tahaei, I. Kobyzev, V. P. Nia, J. J. Clark, and M. Rezagholizadeh, “Krona: Parameter efficient tuning with kronecker adapter,” arXiv preprint arXiv:2212.10650, 2022. — [arXiv:2212.10650](https://arxiv.org/abs/2212.10650) find
  79. X. He, C. Li, P. Zhang, J. Yang, and X. E. Wang, “Parameter-efficient model adaptation for vision transformers,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 1, 2023, pp. 817–825. find
  80. D. J. Kopiczko, T. Blankevoort, and Y. M. Asano, “Vera: Vector-based random matrix adaptation,” arXiv preprint arXiv:2310.11454, 2023. find
  81. S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “Dora: Weight-decomposed low-rank adaptation,” arXiv preprint arXiv:2402.09353, 2024. find
  82. M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi, “Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation,” arXiv preprint arXiv:2210.07558, 2022. find
  83. Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao, “Adaptive budget allocation for parameter-efficient fine-tuning,” arXiv preprint arXiv:2303.10512, 2023. find our card
  84. N. Ding, X. Lv, Q. Wang, Y. Chen, B. Zhou, Z. Liu, and M. Sun, “Sparse low-rank adaptation of pre-trained language models,” arXiv preprint arXiv:2311.11696, 2023. find
  85. S. Haobo, H. Zhao, S. Majumder, and T. Lin, “Increasing model capacity for free: A simple strategy for parameter efficient fine-tuning,” in The Twelfth International Conference on Learning Representations, 2023. find
  86. R. Zhang, R. Qiang, S. A. Somayajula, and P. Xie, “Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning,” arXiv preprint arXiv:2403.09113, 2024. find
  87. A. X. Yang, M. Robeyns, X. Wang, and L. Aitchison, “Bayesian low-rank adaptation for large language models,” arXiv preprint arXiv:2308.13111, 2023. — [arXiv:2308.13111](https://arxiv.org/abs/2308.13111) find
  88. Y. Lin, X. Ma, X. Chu, Y. Jin, Z. Yang, Y. Wang, and H. Mei, “Lora dropout as a sparsity regularizer for overfitting control,” arXiv preprint arXiv:2404.09610, 2024. — [arXiv:2404.09610](https://arxiv.org/abs/2404.09610) find
  89. X. Meng, D. Dai, W. Luo, Z. Yang, S. Wu, X. Wang, P. Wang, Q. Dong, L. Chen, and Z. Sui, “Periodiclora: Breaking the low-rank bottleneck in lora optimization,” arXiv preprint arXiv:2402.16141, 2024. — [arXiv:2402.16141](https://arxiv.org/abs/2402.16141) find
  90. S. Hayou, N. Ghosh, and B. Yu, “Lora+: Efficient low rank adaptation of large models,” arXiv preprint arXiv:2402.12354, 2024. find
  91. T. Wu, J. Wang, Z. Zhao, and N. Wong, “Mixture-of-subspaces in low-rank adaptation,” arXiv preprint arXiv:2406.11909, 2024. find
  92. C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin, “Lorahub: Efficient cross-task generalization via dynamic lora composition,” arXiv preprint arXiv:2307.13269, 2023. find
  93. Q. Liu, X. Wu, X. Zhao, Y. Zhu, D. Xu, F. Tian, and Y. Zheng, “Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications,” arXiv preprint arXiv:2310.18339, 2023. — [arXiv:2310.18339](https://arxiv.org/abs/2310.18339) find
  94. W. Feng, C. Hao, Y. Zhang, Y. Han, and H. Wang, “Mixture-of-loras: An efficient multitask tuning for large language models,” arXiv preprint arXiv:2403.03432, 2024. — [arXiv:2403.03432](https://arxiv.org/abs/2403.03432) find
  95. X. Wu, S. Huang, and F. Wei, “Mixture of lora experts,” arXiv preprint arXiv:2404.13628, 2024. — [arXiv:2404.13628](https://arxiv.org/abs/2404.13628) find
  96. D. Li, Y. Ma, N. Wang, Z. Cheng, L. Duan, J. Zuo, C. Yang, and M. Tang, “Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts,” arXiv preprint arXiv:2404.15159, 2024. find
  97. Y. Mao, L. Mathias, R. Hou, A. Almahairi, H. Ma, J. Han, W.-t. Yih, and M. Khabsa, “Unipelt: A unified framework for parameter-efficient language model tuning,” arXiv preprint arXiv:2110.07577, 2021. find
  98. J. Chen, A. Zhang, X. Shi, M. Li, A. Smola, and D. Yang, “Parameter-efficient fine-tuning design spaces,” arXiv preprint arXiv:2301.01821, 2023. find
  99. Y. Zhang, K. Zhou, and Z. Liu, “Neural prompt search,” 2022. find
  100. H. Zhou, X. Wan, I. Vulić, and A. Korhonen, “Autopeft: Automatic configuration search for parameter-efficient fine-tuning,” arXiv preprint arXiv:2301.12132, 2023. find
  101. Z. Hu, Y. Lan, L. Wang, W. Xu, E.-P. Lim, R. K.-W. Lee, L. Bing, and S. Poria, “Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models,” arXiv preprint arXiv:2304.01933, 2023. find our card
  102. S. Hu, Z. Zhang, N. Ding, Y. Wang, Y. Wang, Z. Liu, and M. Sun, “Sparse structure search for parameter-efficient tuning,” arXiv preprint arXiv:2206.07382, 2022. — [arXiv:2206.07382](https://arxiv.org/abs/2206.07382) find
  103. A. Petrov, P. H. Torr, and A. Bibi, “When do prompting and prefix-tuning work? a theory of capabilities and limitations,” arXiv preprint arXiv:2310.19698, 2023. — [arXiv:2310.19698](https://arxiv.org/abs/2310.19698) find
  104. Y. Wang, J. Chauhan, W. Wang, and C.-J. Hsieh, “Universality and limitations of prompt tuning,” arXiv preprint arXiv:2305.18787, 2023. — [arXiv:2305.18787](https://arxiv.org/abs/2305.18787) find
  105. Y. Choi and J.-H. Lee, “Codeprompt: Task-agnostic prefix tuning for program and language generation,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 5282–5297. find
  106. H. Wu and X. Shi, “Adversarial soft prompt tuning for cross-domain sentiment analysis,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 2438–2447. find
  107. J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” arXiv preprint arXiv:1803.03635, 2018. find
  108. E. Malach, G. Yehudai, S. Shalev-Schwartz, and O. Shamir, “Proving the lottery ticket hypothesis: Pruning is all you need,” in International Conference on Machine Learning. PMLR, 2020, pp. 6682–6691. find
  109. V. Fomenko, H. Yu, J. Lee, S. Hsieh, and W. Chen, “A note on lora,” arXiv preprint arXiv:2404.05086, 2024. — [arXiv:2404.05086](https://arxiv.org/abs/2404.05086) find
  110. A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM journal on imaging sciences, vol. 2, no. 1, pp. 183–202, 2009. find
  111. A. Chambolle, R. A. De Vore, N.-Y. Lee, and B. J. Lucier, “Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage,” IEEE Transactions on image processing, vol. 7, no. 3, pp. 319–335, 1998. find
  112. D. J. MacKay, “A practical bayesian framework for backpropagation networks,” Neural computation, vol. 4, no. 3, pp. 448–472, 1992. find
  113. J. Antorán, D. Janz, J. U. Allingham, E. Daxberger, R. R. Barbano, E. Nalisnick, and J. M. Hernández-Lobato, “Adapting the linearised laplace model evidence for modern deep learning,” in International Conference on Machine Learning. PMLR, 2022, pp. 796–821. find
  114. J. Liu, A. Moreau, M. Preuss, J. Rapin, B. Roziere, F. Teytaud, and O. Teytaud, “Versatile black-box optimization,” in Proceedings of the 2020 Genetic and Evolutionary Computation Conference, 2020, pp. 620–628. find
  115. M. Chen, H. Peng, J. Fu, and H. Ling, “Autoformer: Searching transformers for visual recognition,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 12 270–12 280. find
  116. P. I. Frazier, “A tutorial on bayesian optimization,” arXiv preprint arXiv:1807.02811, 2018. — [arXiv:1807.02811](https://arxiv.org/abs/1807.02811) find
  117. A. Rücklé, G. Geigle, M. Glockner, T. Beck, J. Pfeiffer, N. Reimers, and I. Gurevych, “Adapterdrop: On the efficiency of adapters in transformers,” arXiv preprint arXiv:2010.11918, 2020. find our card
  118. S. He, L. Ding, D. Dong, J. Zhang, and D. Tao, “SparseAdapter: An easy approach for improving the parameter-efficiency of adapters,” in Findings of the Association for Computational Linguistics: EMNLP 2022. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 21… find
  119. L. Hedegaard, A. Alok, J. Jose, and A. Iosifidis, “Structured pruning adapters,” arXiv preprint arXiv:2211.10155, 2022. — [arXiv:2211.10155](https://arxiv.org/abs/2211.10155) find
  120. M. Zhang, C. Shen, Z. Yang, L. Ou, X. Yu, B. Zhuang et al., “Pruning meets low-rank parameter-efficient fine-tuning,” arXiv preprint arXiv:2305.18403, 2023. — [arXiv:2305.18403](https://arxiv.org/abs/2305.18403) find
  121. G. Zeng, P. Zhang, and W. Lu, “One network, many masks: Towards more parameter-efficient transfer learning,” arXiv preprint arXiv:2305.17682, 2023. find
  122. S. Jie, H. Wang, and Z.-H. Deng, “Revisiting the parameter efficiency of adapters from the perspective of precision redundancy,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 17 217–17 226. find
  123. J. Kim, J. H. Lee, S. Kim, J. Park, K. M. Yoo, S. J. Kwon, and D. Lee, “Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization,” arXiv preprint arXiv:2305.14152, 2023. — [arXiv:2305.14152](https://arxiv.org/abs/2305.14152) find
  124. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” arXiv preprint arXiv:2305.14314, 2023. find
  125. Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao, “Loftq: Lora-fine-tuning-aware quantization for large language models,” arXiv preprint arXiv:2310.08659, 2023. — [arXiv:2310.08659](https://arxiv.org/abs/2310.08659) find
  126. H. Guo, P. Greengard, E. P. Xing, and Y. Kim, “Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning,” arXiv preprint arXiv:2311.12023, 2023. find
  127. Y. Xu, L. Xie, X. Gu, X. Chen, H. Chang, H. Zhang, Z. Chen, X. Zhang, and Q. Tian, “Qa-lora: Quantization-aware low-rank adaptation of large language models,” arXiv preprint arXiv:2309.14717, 2023. — [arXiv:2309.14717](https://arxiv.org/abs/2309.14717) find
  128. Y. Chai, J. Gkountouras, G. G. Ko, D. Brooks, and G.-Y. Wei, “Int2. 1: Towards fine-tunable quantized large language models with error correction through low-rank adaptation,” arXiv preprint arXiv:2306.08162, 2023. — [arXiv:2306.08162](https://arxiv.org/abs/2306.08162) find
  129. H. Rajabzadeh, M. Valipour, T. Zhu, M. Tahaei, H. J. Kwon, A. Ghodsi, B. Chen, and M. Rezagholizadeh, “Qdylora: Quantized dynamic low-rank adaptation for efficient large language model tuning,” arXiv preprint arXiv:2402.10462, 2024. find
  130. J. Liu, G. Xiao, K. Li, J. D. Lee, S. Han, T. Dao, and T. Cai, “Bitdelta: Your fine-tune may only be worth one bit,” arXiv preprint arXiv:2402.10193, 2024. find
  131. J. O. Zhang, A. Sax, A. Zamir, L. Guibas, and J. Malik, “Side-tuning: a baseline for network adaptation via additive side networks,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer, 2020, pp. 698–714. find
  132. Y.-L. Sung, J. Cho, and M. Bansal, “Lst: Ladder side-tuning for parameter and memory efficient transfer learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 12 991–13 005, 2022. find
  133. Z. Jiang, C. Mao, Z. Huang, A. Ma, Y. Lv, Y. Shen, D. Zhao, and J. Zhou, “Res-tuning: A flexible and efficient tuning paradigm via unbinding tuner from backbone,” arXiv preprint arXiv:2310.19859, 2023. — [arXiv:2310.19859](https://arxiv.org/abs/2310.19859) find
  134. B. Liao, S. Tan, and C. Monz, “Make your pre-trained model reversible: From parameter to memory efficient fine-tuning,” arXiv preprint arXiv:2306.00477, 2023. — [arXiv:2306.00477](https://arxiv.org/abs/2306.00477) find
  135. L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li, “Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning,” arXiv preprint arXiv:2308.03303, 2023. find
  136. J. Phang, Y. Mao, P. He, and W. Chen, “Hypertuning: Toward adapting large language models without back-propagation,” in International Conference on Machine Learning. PMLR, 2023, pp. 27 854–27 875. find
  137. F. Jin, J. Zhang, and C. Zong, “Parameter-efficient tuning for large language model without calculating its gradients,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 321–330. find
  138. S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora, “Fine-tuning language models with just forward passes,” arXiv preprint arXiv:2305.17333, 2023. find
  139. J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian, “Galore: Memory-efficient llm training by gradient low-rank projection,” arXiv preprint arXiv:2403.03507, 2024. find
  140. Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer et al., “S-lora: Serving thousands of concurrent lora adapters,” arXiv preprint arXiv:2311.03285, 2023. find
  141. T. Zhou and D. Tao, “Godec: Randomized low-rank & sparse matrix decomposition in noisy case,” in Proceedings of the 28th International Conference on Machine Learning, ICML 2011, 2011. find
  142. J. Wright, A. Ganesh, S. Rao, Y. Peng, and Y. Ma, “Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization,” Advances in neural information processing systems, vol. 22, 2009. find
  143. A. N. Gomez, M. Ren, R. Urtasun, and R. B. Grosse, “The reversible residual network: Backpropagation without storing activations,” Advances in neural information processing systems, vol. 30, 2017. find
  144. Y. Huang, Y. Li, Y. Xu, L. Zhang, R. Gan, J. Zhang, and L. Wang, “Mvp-tuning: Multi-view knowledge retrieval with prompt tuning for commonsense reasoning,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 13 417–13 432. find
  145. Z. Zhao, L. Hu, H. Zhao, Y. Shao, and Y. Wang, “Knowledgeable parameter efficient tuning network for commonsense question answering,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 9051– 9063. find
  146. H. Zhao, R. He, M. Xiao, and J. Xu, “Infusing hierarchical guidance into prompt tuning: A parameter-efficient framework for multi-level implicit discourse relation recognition,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023,… find
  147. Y. Ouyang, Y. Cao, Y. Gao, Z. Wu, J. Zhang, and X. Dai, “On prefix-tuning for lightweight out-of-distribution detection,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 1533–1545. find
  148. M. S. Ozdayi, C. Peris, J. Fitzgerald, C. Dupuy, J. Majmudar, H. Khan, R. Parikh, and R. Gupta, “Controlling the extraction of memorized data from large language models via prompt-tuning,” arXiv preprint arXiv:2305.11759, 2023. find
  149. G. Xiao, J. Lin, and S. Han, “Offsite-tuning: Transfer learning without full model,” arXiv preprint arXiv:2302.04870, 2023. — [arXiv:2302.04870](https://arxiv.org/abs/2302.04870) find
  150. T. Che, J. Liu, Y. Zhou, J. Ren, J. Zhou, V. S. Sheng, H. Dai, and D. Dou, “Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization,” arXiv preprint arXiv:2310.15080, 2023. find
  151. Y. Li, M. Du, X. Wang, and Y. Wang, “Prompt tuning pushes farther, contrastive learning pulls closer: A two-stage approach to mitigate social biases,” arXiv preprint arXiv:2307.01595, 2023. find
  152. J. Cho, J. Lei, H. Tan, and M. Bansal, “Unifying vision-and-language tasks via text generation,” in International Conference on Machine Learning. PMLR, 2021, pp. 1931–1942. find
  153. D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592, 2023. find our card
  154. H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” arXiv preprint arXiv:2304.08485, 2023. find our card
  155. S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, “Self-critical sequence training for image captioning,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7008–7024. find our card
  156. Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, “Image captioning with semantic attention,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4651–4659. find
  157. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: Lessons learned from the 2015 mscoco image captioning challenge,” IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 4, pp. 652–663, 2016. find
  158. M. Z. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga, “A comprehensive survey of deep learning for image captioning,” ACM Computing Surveys (CsUR), vol. 51, no. 6, pp. 1–36, 2019. find
  159. P. Wang, Q. Wu, C. Shen, A. Dick, and A. Van Den Hengel, “Fvqa: Fact-based visual question answering,” IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 10, pp. 2413–2427, 2017. find
  160. Q. Wu, D. Teney, P. Wang, C. Shen, A. Dick, and A. Van Den Hengel, “Visual question answering: A survey of methods and datasets,” Computer Vision and Image Understanding, vol. 163, pp. 21–40, 2017. find
  161. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2425– 2433. find
  162. Y.-L. Sung, J. Cho, and M. Bansal, “Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5227–5237. find
  163. Z.-Y. Hu, Y. Li, M. R. Lyu, and L. Wang, “Vl-pet: Vision-and-language parameter-efficient tuning via granularity control,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 3010–3020. find
  164. R. Zhang, J. Han, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, P. Gao, and Y. Qiao, “Llama-adapter: Efficient fine-tuning of language models with zero-init attention,” arXiv preprint arXiv:2303.16199, 2023. find our card
  165. P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue et al., “Llama-adapter v2: Parameter-efficient visual instruction model,” arXiv preprint arXiv:2304.15010, 2023. find
  166. B. Zhao, H. Tu, C. Wei, J. Mei, and C. Xie, “Tuning layernorm in attention: Towards efficient multi-modal llm finetuning,” arXiv preprint arXiv:2312.11420, 2023. find
  167. S. Lee, “Toward continual learning for conversational agents,” arXiv preprint arXiv:1712.09943, 2017. — [arXiv:1712.09943](https://arxiv.org/abs/1712.09943) find
  168. C.-H. Chang, M. Kayed, M. R. Girgis, and K. F. Shaalan, “A survey of web information extraction systems,” IEEE transactions on knowledge and data engineering, vol. 18, no. 10, pp. 1411–1428, 2006. find
  169. W. Yang, Y. Xie, A. Lin, X. Li, L. Tan, K. Xiong, M. Li, and J. Lin, “End-to-end open-domain question answering with bertserini,” arXiv preprint arXiv:1902.01718, 2019. find our card
  170. J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017. find
  171. A. Madotto, Z. Lin, Z. Zhou, S. Moon, P. Crook, B. Liu, Z. Yu, E. Cho, and Z. Wang, “Continual learning in task-oriented dialogue systems,” arXiv preprint arXiv:2012.15504, 2020. find
  172. Q. Zhu, B. Li, F. Mi, X. Zhu, and M. Huang, “Continual prompt tuning for dialog state tracking,” arXiv preprint arXiv:2203.06654, 2022. find
  173. Y. Dai, H. Lang, Y. Zheng, F. Huang, L. Si, and Y. Li, “Lifelong learning for question answering with hierarchical prompts,” arXiv preprint arXiv:2208.14602, 2022. — [arXiv:2208.14602](https://arxiv.org/abs/2208.14602) find
  174. Z. Liang, F. Wei, Y. Jie, Y. Qian, Z. Hao, and B. Han, “Prompts can play lottery tickets well: Achieving lifelong information extraction via lottery prompt tuning,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 277–292. find
  175. X. Wang, T. Chen, Q. Ge, H. Xia, R. Bao, R. Zheng, Q. Zhang, T. Gui, and X. Huang, “Orthogonal subspace learning for language model continual learning,” arXiv preprint arXiv:2310.14152, 2023. find
  176. S. Chen, S. Wong, L. Chen, and Y. Tian, “Extending context window of large language models via positional interpolation,” arXiv preprint arXiv:2306.15595, 2023. find
  177. Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia, “Longlora: Efficient fine-tuning of long-context large language models,” arXiv preprint arXiv:2309.12307, 2023. find our card
  178. J. Yang, “Longqlora: Efficient and effective method to extend context length of large language models,” arXiv preprint arXiv:2311.04879, 2023. find
  179. S. Tan, X. Li, S. Patil, Z. Wu, T. Zhang, K. Keutzer, J. E. Gonzalez, and R. A. Popa, “Lloco: Learning long contexts offline,” arXiv preprint arXiv:2404.07979, 2024. find
  180. Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett et al., “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” Advances in Neural Information Processing Systems, vol. 36, 2024. find
  181. Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” in International Conference on Machine Learning. PMLR, 2023, pp. 31 094–31 116. find our card
  182. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,” Advances in Neural Information Processing Systems, vol. 35, pp. 30 318–30 332, 2022. find
  183. H. Kang, Q. Zhang, S. Kundu, G. Jeong, Z. Liu, T. Krishna, and T. Zhao, “Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm,” arXiv preprint arXiv:2403.05527, 2024. — [arXiv:2403.05527](https://arxiv.org/abs/2403.05527) find
  184. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale. arxiv 2020,” arXiv preprint arXiv:2010.11929, 2010. find
  185. A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer, “How to train your vit? data, augmentation, and regularization in vision transformers,” arXiv preprint arXiv:2106.10270, 2021. — [arXiv:2106.10270](https://arxiv.org/abs/2106.10270) find
  186. X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised vision transformers,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9640–9649. find
  187. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 000–16 009. find our card
  188. M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin et al., “Scaling vision transformers to 22 billion parameters,” in International Conference on Machine Learning. PMLR, 2023, pp. 7480–7512. find
  189. Z. Chen, Y. Duan, W. Wang, J. He, T. Lu, J. Dai, and Y. Qiao, “Vision transformer adapter for dense predictions,” arXiv preprint arXiv:2205.08534, 2022. — [arXiv:2205.08534](https://arxiv.org/abs/2205.08534) find
  190. Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister, “Learning to prompt for continual learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 139–149. find
  191. Q. Gao, C. Zhao, Y. Sun, T. Xi, G. Zhang, B. Ghanem, and J. Zhang, “A unified continual learning framework with general parameter-efficient tuning,” arXiv preprint arXiv:2303.10070, 2023. — [arXiv:2303.10070](https://arxiv.org/abs/2303.10070) find
  192. L. Ren, C. Chen, L. Wang, and K. Hua, “Learning semantic proxies from visual prompts for parameter-efficient fine-tuning in deep metric learning,” arXiv preprint arXiv:2402.02340, 2024. — [arXiv:2402.02340](https://arxiv.org/abs/2402.02340) find
  193. M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” in European Conference on Computer Vision. Springer, 2022, pp. 709–727. find
  194. S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo, “Adaptformer: Adapting vision transformers for scalable visual recognition,” Advances in Neural Information Processing Systems, vol. 35, pp. 16 664–16 678, 2022. find
  195. S. Jie and Z.-H. Deng, “Convolutional bypasses are better vision transformer adapters,” arXiv preprint arXiv:2207.07039, 2022. — [arXiv:2207.07039](https://arxiv.org/abs/2207.07039) find
  196. S. Yoo, E. Kim, D. Jung, J. Lee, and S. Yoon, “Improving visual prompt tuning for self-supervised vision transformers,” arXiv preprint arXiv:2306.05067, 2023. — [arXiv:2306.05067](https://arxiv.org/abs/2306.05067) find
  197. J. Pan, Z. Lin, X. Zhu, J. Shao, and H. Li, “St-adapter: Parameter-efficient image-to-video transfer learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 26 462–26 477, 2022. find
  198. T. Yang, Y. Zhu, Y. Xie, A. Zhang, C. Chen, and M. Li, “Aim: Adapting image models for efficient video action recognition,” arXiv preprint arXiv:2302.03024, 2023. — [arXiv:2302.03024](https://arxiv.org/abs/2302.03024) find
  199. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763. find our card
  200. C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in International conference on machine learning. PMLR, 2021, pp. 4904–4916. find our card
  201. Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, “Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm,” arXiv preprint arXiv:2110.05208, 2021. — [arXiv:2110.05208](https://arxiv.org/abs/2110.05208) find
  202. A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela, “Flava: A foundational language and vision alignment model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15 638–15 650. find
  203. M. Xu, Z. Zhang, F. Wei, H. Hu, and X. Bai, “Side adapter network for open-vocabulary semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2945–2954. find
  204. Q. Yu, J. He, X. Deng, X. Shen, and L.-C. Chen, “Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,” arXiv preprint arXiv:2308.02487, 2023. — [arXiv:2308.02487](https://arxiv.org/abs/2308.02487) find
  205. Z. Xu, Z. Chen, Y. Zhang, Y. Song, X. Wan, and G. Li, “Bridging vision and language encoders: Parameter-efficient tuning for referring image segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 17 503–17 512. find
  206. R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao, P. Gao, and H. Li, “Pointclip: Point cloud understanding by clip,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8552–8562. find
  207. X. Zhu, R. Zhang, B. He, Z. Guo, Z. Zeng, Z. Qin, S. Zhang, and P. Gao, “Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2639–2650. find
  208. Z. Wang, X. Yu, Y. Rao, J. Zhou, and J. Lu, “P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting,” Advances in neural information processing systems, vol. 35, pp. 14 388–14 402, 2022. find
  209. T. Huang, B. Dong, Y. Yang, X. Huang, R. W. Lau, W. Ouyang, and W. Zuo, “Clip2point: Transfer clip to point cloud classification with image-depth pre-training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22 157–22 167. find
  210. C. Ju, T. Han, K. Zheng, Y. Zhang, and W. Xie, “Prompting visual-language models for efficient video understanding,” in European Conference on Computer Vision. Springer, 2022, pp. 105–124. find
  211. B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng, J. Fu, S. Xiang, and H. Ling, “Expanding language-image pretrained models for general video recognition,” in European Conference on Computer Vision. Springer, 2022, pp. 1–18. find
  212. Z. Lin, S. Geng, R. Zhang, P. Gao, G. de Melo, X. Wang, J. Dai, Y. Qiao, and H. Li, “Frozen clip models are efficient video learners,” in European Conference on Computer Vision. Springer, 2022, pp. 388–404. find
  213. Z. Han, F. Zhu, Q. Lao, and H. Jiang, “Zero-shot referring expression comprehension via structural similarity between images and captions,” arXiv preprint arXiv:2311.17048, 2023. — [arXiv:2311.17048](https://arxiv.org/abs/2311.17048) find
  214. S. Doveh, A. Arbelle, S. Harary, E. Schwartz, R. Herzig, R. Giryes, R. Feris, R. Panda, S. Ullman, and L. Karlinsky, “Teaching structured vision & language concepts to vision & language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2657–2668… find
  215. S. Nag, X. Zhu, Y.-Z. Song, and T. Xiang, “Zero-shot temporal action detection via vision-language prompting,” in European Conference on Computer Vision. Springer, 2022, pp. 681–697. find
  216. K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” International Journal of Computer Vision, vol. 130, no. 9, pp. 2337–2348, 2022. find
  217. ——, “Conditional prompt learning for vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 816–16 825. find
  218. B. Zhu, Y. Niu, Y. Han, Y. Wu, and H. Zhang, “Prompt-aligned gradient for prompt tuning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 659–15 669. find
  219. M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan, “Maple: Multi-modal prompt learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19 113– 19 122. find
  220. M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao, “Test-time prompt tuning for zero-shot generalization in vision-language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 14 274–14 289, 2022. find
  221. C.-M. Feng, K. Yu, Y. Liu, S. Khan, and W. Zuo, “Diverse data augmentation with diffusions for effective test-time prompt tuning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2704–2714. find
  222. P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao, “Clip-adapter: Better vision-language models with feature adapters,” International Journal of Computer Vision, pp. 1–15, 2023. find
  223. R. Zhang, R. Fang, W. Zhang, P. Gao, K. Li, J. Dai, Y. Qiao, and H. Li, “Tip-adapter: Training-free clip-adapter for better vision-language modeling,” arXiv preprint arXiv:2111.03930, 2021. — [arXiv:2111.03930](https://arxiv.org/abs/2111.03930) find
  224. E. Orhan, “A simple cache model for image recognition,” Advances in Neural Information Processing Systems, vol. 31, 2018. find
  225. E. Grave, M. M. Cisse, and A. Joulin, “Unbounded cache model for online language modeling with open vocabulary,” Advances in neural information processing systems, vol. 30, 2017. find
  226. J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020. find
  227. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in International conference on machine learning. PMLR, 2015, pp. 2256–2265. find our card
  228. Z. Han, Y. Wang, L. Zhou, P. Wang, B. Yan, J. Zhou, Y. Wang, and D. Shen, “Contrastive diffusion model with auxiliary guidance for coarse-to-fine pet reconstruction,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 239–249. find
  229. L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang, “Diffusion models: A comprehensive survey of methods and applications,” ACM Computing Surveys, vol. 56, no. 4, pp. 1–39, 2023. find
  230. F.-A. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vision: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. find
  231. P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” Advances in neural information processing systems, vol. 34, pp. 8780–8794, 2021. find
  232. N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22 500–22 510. find
  233. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695. find
  234. S. Luo, Y. Tan, S. Patil, D. Gu, P. von Platen, A. Passos, L. Huang, J. Li, and H. Zhao, “Lcm-lora: A universal stable-diffusion acceleration module,” arXiv preprint arXiv:2311.05556, 2023. find
  235. W. Chai, D. Zheng, J. Cao, Z. Chen, C. Wang, and C. Ma, “Speedupnet: A plug-and-play hyper-network for accelerating text-to-image diffusion models,” arXiv preprint arXiv:2312.08887, 2023. — [arXiv:2312.08887](https://arxiv.org/abs/2312.08887) find
  236. J. Z. Wu, Y. Ge, X. Wang, S. W. Lei, Y. Gu, Y. Shi, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou, “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 7623–7633. find
  237. Z. Xing, Q. Dai, H. Hu, Z. Wu, and Y.-G. Jiang, “Simda: Simple diffusion adapter for efficient video generation,” arXiv preprint arXiv:2308.09710, 2023. — [arXiv:2308.09710](https://arxiv.org/abs/2308.09710) find
  238. B. Zeng, S. Li, Y. Feng, H. Li, S. Gao, J. Liu, H. Li, X. Tang, J. Liu, and B. Zhang, “Ipdreamer: Appearance-controllable 3d object generation with image prompts,” arXiv preprint arXiv:2310.05375, 2023. — [arXiv:2310.05375](https://arxiv.org/abs/2310.05375) find
  239. J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al., “Flamingo: a visual language model for few-shot learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 23 716–23 736, 2022. find our card
  240. L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 3836–3847. find
  241. R. Gandikota, J. Materzynska, T. Zhou, A. Torralba, and D. Bau, “Concept sliders: Lora adaptors for precise control in diffusion models,” arXiv preprint arXiv:2311.12092, 2023. — [arXiv:2311.12092](https://arxiv.org/abs/2311.12092) find
  242. C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi, Y. Shan, and X. Qie, “T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models,” arXiv preprint arXiv:2302.08453, 2023. — [arXiv:2302.08453](https://arxiv.org/abs/2302.08453) find
  243. R. Gal, Y. Alaluf, Y. Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-Or, “An image is worth one word: Personalizing text-to-image generation using textual inversion,” arXiv preprint arXiv:2208.01618, 2022. — [arXiv:2208.01618](https://arxiv.org/abs/2208.01618) find
  244. N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J.-Y. Zhu, “Multi-concept customization of text-to-image diffusion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1931–1941. find
  245. H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang, “Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models,” arXiv preprint arXiv:2308.06721, 2023. — [arXiv:2308.06721](https://arxiv.org/abs/2308.06721) find
  246. OpenAI, “Gpt-4,” in https://openai.com/gpt-4, 2023. ↗
  247. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023. find our card
  248. Z. Zhou, X. Wei, J. Zhang, and G. Sun, “tPetSu: A unified framework for tParameter-Efficientu transformers serving,” in 2022 USENIX Annual Technical Conference (USENIX ATC 22), 2022, pp. 489–504. find
  249. B. Wu, R. Zhu, Z. Zhang, P. Sun, X. Liu, and X. Jin, “tdLoRAu: Dynamically orchestrating requests and adapters for tLoRAutLLMu serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 911–927. find
  250. C. Gao and S. Q. Zhang, “Dlora: Distributed parameter-efficient fine-tuning solution for large language model,” arXiv preprint arXiv:2404.05182, 2024. find
  251. G. Xiao, J. Lin, and S. Han, “Offsite-tuning: Transfer learning without full model,” arXiv preprint arXiv:2302.04870, 2023. — [arXiv:2302.04870](https://arxiv.org/abs/2302.04870) find
  252. L. Chen, Z. Ye, Y. Wu, D. Zhuo, L. Ceze, and A. Krishnamurthy, “Punica: Multi-tenant lora serving,” arXiv preprint arXiv:2310.18547, 2023. find
  253. S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan, “Peft: State-of-the-art parameter-efficient fine-tuning methods,” https://github.com/huggingface/peft, 2022. ↗
  254. C. Poth, H. Sterz, I. Paul, S. Purkayastha, L. Engländer, T. Imhof, I. Vulić, S. Ruder, I. Gurevych, and J. Pfeiffer, “Adapters: A unified library for parameter-efficient and modular transfer learning,” 2023. find
  255. K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, Z. Zhang, D. Cheng, C. Zhu, T. Cheng, Q. Zhao, B. Li, X. Lu, R. Zhu, Y. Wu, J. Dai, J. Wang, J. Shi, W. Ouyang, C. C. Loy, and D. Lin, “MMDetection: Open mmlab detection toolbox and benchmark,” arXiv preprint arXiv:1… — [arXiv:1906.07155](https://arxiv.org/abs/1906.07155) find
  256. S. Q. Zhang, T. Tambe, N. Cuevas, G.-Y. Wei, and D. Brooks, “Camel: Co-designing ai models and embedded drams for efficient on-device learning,” arXiv preprint arXiv:2305.03148, 2023. — [arXiv:2305.03148](https://arxiv.org/abs/2305.03148) find
  257. T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh, “Video generation models as world simulators,” 2024. [Online]. Available: https: //openai.com/research/video-generation-models-as-world-simulators find
  258. A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023. find
  259. Y. Bai, X. Geng, K. Mangalam, A. Bar, A. Yuille, T. Darrell, J. Malik, and A. A. Efros, “Sequential modeling enables scalable learning for large vision models,” arXiv preprint arXiv:2312.00785, 2023. — [arXiv:2312.00785](https://arxiv.org/abs/2312.00785) find
  260. A. Dosovitskiy and T. Brox, “Inverting visual representations with convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4829–4837. find
  261. Z. He, T. Zhang, and R. B. Lee, “Model inversion attacks against collaborative inference,” in Proceedings of the 35th Annual Computer Security Applications Conference, 2019, pp. 148–162. find