Elamur Research⌕
☰
home / papers / 2026 / GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning

GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning

Paper · 2026 · Kaiyuan Tian, Linbo Qiao, Yu Tang et al.

Summary

Full-parameter fine-tuning of large language models needs a lot of GPU memory. LoRA-style methods reduce this cost but lose expressiveness. Layer-wise methods such as LISA reduce memory by training only a few layers at a time. However, they choose those layers with fixed rules that ignore how layer relevance changes between tasks and over the course of training.

GRASS addresses this by scoring each layer with its mean gradient norm (MGN). The scores are refreshed periodically with an exponential moving average and turned into sampling probabilities, so the most influential layers are trained in each period while the rest stay frozen. To save more memory, optimizer states are offloaded to the CPU layer by layer, and these transfers overlap with computation.

The method was tested on TinyLlama, Gemma-2B and LLaMA2-7B using arithmetic and commonsense reasoning benchmarks. GRASS beat existing approaches by up to 4.38 average accuracy points and used up to 19.97% less memory. This makes full-capacity fine-tuning more practical on limited hardware.

Key points

  • A motivating comparison on arithmetic reasoning (MultiArith, AddSub, GSM8K, AQuA, SingleEq, SVAMP) shows LISA averaging 56.57 versus 60.46 for full fine-tuning. The drops are largest on GSM8K and SingleEq.
  • MGN profiles on TinyLlama vary by task. For example, layer 20 scores high for commonsense reasoning but not for arithmetic reasoning.
  • An ablation tested GRASS without adaptive probability updates. Adaptive sampling raised average arithmetic accuracy:
    • TinyLlama: from 30.69 to 34.22
    • Gemma-2B: from 59.00 to 60.65
    • LLaMA2-7B: from 56.59 to 59.59
  • Hyperparameter and robustness findings:
    • Sampling periods of 25–50 steps worked best.
    • Training more active layers per period generally helped.
    • Accuracy varied by at most 0.24 points across random seeds.
  • Efficiency on LLaMA2-7B:
    • Overlapping offload with computation increased throughput by 1.08×, bringing it in line with LoRA.
    • The probing and MGN computations add roughly 2–6% to training time, and only 2.01% on LLaMA2-7B.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of Defense TechnologyChangsha · China10 works · 931 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Kaiyuan Tian, Linbo Qiao, Yu Tang, Gongqingjian Jiang, Baihui Liu, Yifu Gao, Xialin Su and Dongsheng Li. 2026. GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.475

@inproceedings{tian2026grass,
  title = {{GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning}},
  author = {Kaiyuan Tian, Linbo Qiao, Yu Tang, Gongqingjian Jiang, Baihui Liu, Yifu Gao, Xialin Su and Dongsheng Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.475},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.475}
}

References

42 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  3. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45.
  4. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL-HLT 2019, pages 2924–2936. Association for Computational Linguistics.
  5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, and Matthias Plappert. 2021. Training verifiers to solve math word problems.
  7. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, and Shengding Hu. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint, arXiv:2203.06904. — [arXiv:2203.06904](https://arxiv.org/abs/2203.06904) find
  8. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations.
  9. Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In EMNLP 2014, pages 523–533. Association for Computational Linguistics.
  10. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, and Mona Attariyan. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. find
  11. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. find
  12. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Lu Wang. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  13. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP 2018, pages 2381–2391. Association for Computational Linguistics. find
  14. Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, and Xing Xu. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In EMNLP 2023, pages 5254–5276. Association for Computational Linguistics.
  15. Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024. Lisa: Layer-wise importance sampling for memory-efficient large language model fine-tuning. Advances in Neural Information Processing Systems, 37:57018–57049. find
  16. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597. find
  17. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP 2021, pages 3045–3059. Association for Computational Linguistics.
  18. Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Pretrained language model for text generation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4492–4499. International Joint Conferences on Artificial Intelligence Orga… find
  19. Pengxiang Li, Lu Yin, Xiaowei Gao, and Shiwei Liu. 2025. Outlier-weighed layerwise sampling for llm fine-tuning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19460– 19473. find
  20. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL-IJCNLP 2021, pages 4582–4597. Association for Computational Linguistics. find
  21. Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. 2023. Relora: High-rank training through low-rank updates. In The Twelfth International Conference on Learning Representations.
  22. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In NAACL-HLT 2021, pages 2080–2094. Association for Computational Linguistics.
  23. Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In EMNLP 2015, pages 1743– 1752. Association for Computational Linguistics.
  24. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106. find
  25. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Commonsense reasoning about social interactions. In EMNLP-IJCNLP 2019, pages 4463–4473. Association for Computational Linguistics. find
  26. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and 1 others. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. find
  27. Kaiyuan Tian, Linbo Qiao, Baihui Liu, Gongqingjian Jiang, Shanshan Li, and Dongsheng Li. 2026. A survey on memory-efficient transformer-based model training in ai for science. Frontiers of Computer Science, 20(11):2011355. find
  28. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In ACL 2017, pages 158–167. Association for Computational Linguistics. find
  29. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  30. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, pages 32100–32121. PMLR. find
  31. Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2023. Dylora: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In EACL 2023, pages 3274–3287. Association for Computational Linguistics.
  32. Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL 2022, pages 61–68. Association for Computational Linguistics.
  33. Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, and Fei Yang. 2025. Parameter-efficient fine-tuning in large language models: A survey of methodologies. Artificial Intelligence Review, 58(8):227. find
  34. Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2022. Adamix: Mixture-of-adaptations for parameter-efficient model tuning. In EMNLP 2022, pages 5744–5760. Association for Computational Linguistics.
  35. Wenhan Xia, Chengwei Qin, and Elad Hazan. 2024. Chain of lora: Efficient fine-tuning of language models via residual learning. arXiv preprint arXiv:2401.04151. — [arXiv:2401.04151](https://arxiv.org/abs/2401.04151) find
  36. Kai Yao, Penglei Gao, Lichun Li, Yuan Zhao, Xiaofeng Wang, Wei Wang, and Jianke Zhu. 2024. Layer-wise importance matters: Less memory for better performance in parameter-efficient fine-tuning of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 19… find
  37. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In ACL 2019, pages 4791–4800. Association for Computational Linguistics. find
  38. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385.
  39. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. Galore: Memory-efficient llm training by gradient low-rank projection. In Forty-first International Conference on Machine Learning. find
  40. Peilin Zhao and Tong Zhang. 2015. Stochastic optimization with importance sampling for regularized loss minimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1–9. PMLR. find
  41. Ligeng Zhu, Lanxiang Hu, Ji Lin, and Song Han. 2023. Lift: Efficient layer-wise fine-tuning for large model models. find
  42. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models: Empirical results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2765–2781, Mexi…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2026-grass-gradient-based-adaptive-layer/