Elamur Research⌕
☰
home / papers / 2026 / Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning

Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning

Paper · 2026 · JuneYoung Park, Yuri Hong, Seongwan Kim et al.

Summary

The paper targets a practical obstacle to personalizing language models on phones and laptops: even with LoRA, the activations kept for backpropagation can exceed what a device can spare. Current options force a compromise. MeBP gives exact gradients but relies on automatic differentiation, which keeps more tensors than needed. MeZO uses little memory, but its gradients are noisy. The authors propose MeSP (Memory-efficient Structured Backpropagation). It uses hand-written backward passes for each transformer component, keeps only block outputs during the forward pass, and recomputes intermediates one layer at a time. The key trick is that the LoRA projection h = xA is very narrow (rank 8–32), so recomputing it is cheaper than caching it. Experiments use Qwen2.5 models (0.5B–3B) on WikiText-2, implemented in MLX on Apple Silicon. MeSP produces the same gradients as MeBP while using 42–62% less peak memory, at roughly 28% extra compute time. This makes exact first-order fine-tuning feasible in memory budgets where it was previously impractical.

Key points

  • Peak memory at sequence length 256: on Qwen2.5-0.5B, MeSP drops peak memory from 360.8 MB (MeBP) to 136.2 MB, a 62% cut. The savings shrink as models grow, falling to 49% for 1.5B and 42% for 3B, because weights dominate memory in larger models.
  • Scaling with sequence length and rank: MeSP saves 51–62% across sequence lengths, while MeBP grows from about 253 MB to 1050 MB. Savings stay near 60–63% across LoRA ranks. By contrast, MeZO actually uses more memory than MeBP at rank 32.
  • Training quality: over 100K steps, MeSP and MeBP reach the same final loss (about 2.6). MeZO ends roughly 22% higher (about 3.18).
  • Why MeZO is slow: its gradient estimates barely align with the true gradients, with cosine similarity of about 0.001 and sign agreement near 50%.
  • Ablation and code: on Qwen2.5-3B, recomputing h instead of storing it saves an extra 7.6% memory for 6.2% more time. The code is at github.com/crinex/acl-mesp.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as JuneYoung Park, Yuri Hong, Seongwan Kim and Jaeho Lee. 2026. Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.62

@inproceedings{park2026memory,
  title = {{Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning}},
  author = {JuneYoung Park, Yuri Hong, Seongwan Kim and Jaeho Lee},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.62},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.62}
}

References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Apple Machine Learning Research. 2023. MLX: An efficient machine learning framework for apple silicon. https://github.com/ml-explore/mlx. ↗
  2. In Advances in Neural Information Processing Systems. find
  3. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems.
  4. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. find
  5. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  6. Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. 2022. On-device training under 256kb memory. In Advances in Neural Information Processing Systems. find
  7. Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. 2023. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems. find
  8. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
  9. Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  10. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
  11. Yue Xu, Yuxin Chen, Ligeng Zhu, Ji Lin, and Song Han. 2023. PockEngine: Sparse and efficient fine-tuning in a pocket. arXiv preprint arXiv:2310.17752. find
  12. Apple Machine Learning Research. 2024. Memory-efficient backpropagation for on-device LLM fine-tuning. https://github.com/apple/ ml-mebp. ↗
  13. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. find
  14. Tri Dao. 2023. FlashAttention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.
  15. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/park-2026-memory-efficient-structured-backpropagation-device/