Liger: Linearizing Large Language Models to Gated Recurrent Structures

Paper · 2025 · Disen Lan, Weigao Sun, Jiaxi Hu et al.

Summary

Transformer LLMs have attention cost that grows quadratically with sequence length, and their KV cache keeps growing during generation. Linear recurrent models avoid this with linear-time training and fixed-memory decoding, but pretraining them from scratch is expensive and risky. Earlier linearization methods (SUPRA, MambaInLlama, LoLCATs) convert pretrained Transformers instead. However, they add feature-map or gate modules that must be trained from zero, and they mostly ignore gating. Liger converts a pretrained LLM into a gated linear recurrent model without adding parameters. The existing key projection keeps its usual role and also produces gate values through a parameter-free pooling step. The converted model is then briefly fine-tuned with LoRA on an autoregressive objective. The authors also introduce Liger Attention, which combines sliding-window softmax attention with gated linear recurrence inside a single layer. On Llama-3 models from 1B to 8B, Liger keeps about 93% of the original quality using roughly 0.02% of the pretraining tokens, and it outperforms prior linearization approaches. This makes efficient recurrent LLMs practical without full pretraining.

Key points

Gate construction: gates are computed from the pretrained key projection W_K via pooling, so no new gate parameters are trained. Ablations show this is competitive with randomly initialized gate projections, and that removing LoRA, GLA or SWA hurts results.
Scale and cost: Liger was tested on Llama-3 at 1B, 3B and 8B. It recovers about 93% of base performance at about 0.02% of pretraining tokens. The gap to the original model shrinks from 4.8% at 1B to 1.8% at 8B. By comparison, GLA-1B trained from scratch averages only 46.9%.
Generality: besides GLA, the method also works with other gated recurrent variants, HGRN2 and GSA.
Efficiency: Liger-GLA-8B was compared with Llama-3-8B, both with and without Flash-Attention-2, on a single A800 80GB GPU. The setup used batch size 16, a 128-token prefix and decoding lengths from 1K to 32K. Liger shows linear-time decoding with constant memory.
Resources: the paper appeared at ICML 2025. Code is at github.com/OpenSparseLLMs/Linearization, and models are in the Hugging Face collection linear-moe-hub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields, topics and concepts

📚 Shelves 5Computer Science · 16,667 works Artificial Intelligence · 16,194 works Arts and Humanities · 16,039 works Language and Linguistics · 15,891 works Computer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Language models & pretraining · 3,320 works

Cite

Cite as Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du and Yu Cheng. 2025. Liger: Linearizing Large Language Models to Gated Recurrent Structures. arXiv:2503.01496.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.01496 · arXiv 2503.01496

@misc{lan2025liger,
  title = {{Liger: Linearizing Large Language Models to Gated Recurrent Structures}},
  author = {Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du and Yu Cheng},
  year = {2025},
  doi = {10.48550/arxiv.2503.01496},
  eprint = {2503.01496},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.01496}
}

References

2 references, 2 of them with a card of ours. ↗ leads to the work.

Aghajanyan, A., Zettlemoyer, L., and Gupta, S. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020. URL https://arxiv.org/abs/2012.13255. ↗
Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Dylan Zinsley, J. Z., Rudra, A., and Ré, C. Simple linear attention language models balance the recall-throughput tradeoff. arXiv preprint arXiv:2402.18668, 2024. \#\# 6. Conclusion

Updated 30 October 2026

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lan-2025-liger-linearizing-large-language-models/