Elamur Research⌕
☰
home / papers / 2026 / Acceleration of Backpropagation in Linear Layers of Transformer Models Based on Gradient Structure

Acceleration of Backpropagation in Linear Layers of Transformer Models Based on Gradient Structure

Paper · 2026 · Dmitrii Topchii, Alexander Panchenko and Viktoriia A. Chekalina

Summary

Fine-tuning Transformers spends much of its compute on the backward pass through linear layers. Many NLP tasks use short inputs padded to a fixed context length, and the authors show this padding creates structured row sparsity in the output gradients. They measured this in BERT-base and RoBERTa-base on GLUE (short) versus WikiText-103 (full-length), and in LLaMA 3.2 3B on OASST1 and reasoning datasets. For LLaMA, near-zero gradient noise is removed with a threshold found from a knee point in the gradient-value distribution.

Their method, Sparsity-Exploiting Backward Pass (SEBP), keeps only a fixed number of the strongest gradient rows. A custom Triton kernel then turns the large sparse–dense multiplication into a smaller dense–dense one, and the original layout is restored afterwards. Backward passes run roughly twice as fast, memory stays close to plain PyTorch, and convergence and accuracy are preserved. This makes SEBP an option for memory-limited or on-device fine-tuning, where DeepSpeed-style speedups cost too much memory.

Key points

  • Reported backward-pass speedups are about 2.15× for BERT-base on GLUE and 1.99× for Llama-3.2-3B on reasoning benchmarks, measured on A100 GPUs.
  • Memory use stays nearly the same as standard PyTorch fine-tuning. DeepSpeed is faster overall but consumes noticeably more memory.
  • For LLaMA, the noise-truncation threshold is ϵ = 3.98·10⁻⁵, chosen from the knee of a smoothed CDF of gradient values.
  • After fine-tuning Llama-3.2-3B, SEBP improved ARC-C, ARC-E, HellaSwag and Winogrande by roughly +0.008 to +0.013. DeepSpeed slightly lowered both ARC scores.
  • SEBP extends SparseGrad, and its code is available at github.com/Dmitrii-Topchii/SEBP.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Dmitrii Topchii, Alexander Panchenko and Viktoriia A. Chekalina. 2026. Acceleration of Backpropagation in Linear Layers of Transformer Models Based on Gradient Structure. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-srw.31

@inproceedings{topchii2026acceleration,
  title = {{Acceleration of Backpropagation in Linear Layers of Transformer Models Based on Gradient Structure}},
  author = {Dmitrii Topchii, Alexander Panchenko and Viktoriia A. Chekalina},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.31},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.31}
}

References

1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. \#\# Limitations Viktoriia A. Chekalina, Anna Rudenko, Gleb Mezentsev, Aleksandr Mikhalev, Alexander Panchenko, and Ivan Oseledets. 2024. SparseGrad: A selective method for efficient fine-tuning of MLP layers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p… find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/topchii-2026-acceleration-backpropagation-linear-layers-transformer/