Elamur Research⌕
☰
home / papers / 2026 / FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression

FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression

Paper · 2026 · Jiayi Tian, Ryan Solgi, Jinming Lu et al.

Summary

Large language models are costly to run on limited hardware. Low-rank compression helps, but SVD-based variants often lose accuracy, need recovery fine-tuning, and split layers into two smaller matrix multiplications that run poorly on GPUs. SliceGPT avoids SVD but adds adapters on residual paths, which costs memory. FLAT-LLM is a training-free structural compression method that works in the activation space. For each attention head, it runs PCA on the value-layer outputs using calibration samples and keeps only the leading eigenvectors. Those truncated bases are then folded into the value and output projection weights, which shrinks both with no extra modules. Query and key projections are compressed separately with the same per-head procedure. A greedy, tuning-free scheme called Importance-Preserving Rank Selection (IPRS) shifts rank budgets between decoder layers. Calibration takes only minutes. Across 5 models (LLaMA and Mistral families) and 11 datasets, FLAT-LLM beats structural pruning baselines and runs faster than low-rank competitors. This makes it a practical option for deploying compressed models.

Key points

  • The core idea is head-wise PCA on value outputs, with eigenvectors absorbed jointly into the value and output weights. It adds no memory overhead, the compression ratio equals the retained rank ratio, and it is compatible with GQA.
  • IPRS completes in seconds, is over 100× faster than the hypernetwork-based Adaptive SVD, and outperforms MoDeGPT's allocation. On LLaMA-2 13B at 40% compression, it beats uniform ranks by 8.0% (WikiText-2 calibration) and 5.4% (Alpaca calibration).
  • On LLaMA-2 7B, throughput is at least 1.50× that of the original model at every ratio tested, and 1.68× even at 10% compression. Against SliceGPT and SVD-LLM, throughput is up to 1.63× and 1.93× higher, with 19% and 20% less memory.
  • Combining 30% FLAT-LLM compression with GPTQ 3/4-bit gives lower WikiText-2 perplexity than GPTQ 2/3-bit alone at a similar or smaller size. Overall compression reaches about 5–8×.
  • The authors note limitations: real speedups still need CUDA kernel–level optimization, and the rank allocation is not hardware-aware. Code is available at github.com/TTTTTTris/FLAT-LLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, Santa BarbaraSanta Barbara · United States57 works · 9,752 citations🏢IntelSanta Clara · United States15 works · 41,257 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Latent semantic indexing · 29 works

Cite

Cite as Jiayi Tian, Ryan Solgi, Jinming Lu, Yifan Yang, Hai Li and Zheng Zhang. 2026. FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.156

@inproceedings{tian2026flat,
  title = {{FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression}},
  author = {Jiayi Tian, Ryan Solgi, Jinming Lu, Yifan Yang, Hai Li and Zheng Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.156},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.156}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895– 4…
  2. Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865–10873. find
  3. Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations. find
  4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  5. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  6. Shangqian Gao, Ting Hua, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2024b. Adaptive rank selections for low-rank approximation of language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum… find
  7. Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. 2024. Billm: pushing the limit of post-training quantization for llms. In Proceedings of the 41st International Conference on Machine Learning, pages 20023–20042. find
  8. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  9. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174.
  10. Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. 2024. Modegpt: Modular decomposition for large language model compression. arXiv preprint arXiv:2408.09632. find
  11. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  12. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720.
  13. Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853.
  14. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations. find
  15. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilebert: a compact task-agnostic bert for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158–2170.
  16. Jiayi Tian, Chao Fang, Haonan Wang, and Zhongfeng Wang. 2023. Bebert: Efficient and robust binary ensemble bert. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  17. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  18. Tycho FA van der Ouderaa, Markus Nagel, Mart Van Baalen, and Tijmen Blankevoort. 2024. The llm surgeon. In The Twelfth International Conference on Learning Representations. find
  19. Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. 2024. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378. — [arXiv:2403.07378](https://arxiv.org/abs/2403.07378) find
  20. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and 1 others. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  21. Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, and 1 others. 2025. Wanda++: Pruning large language models via regional gradients. arXiv preprint arXiv:2503.04992. find
  22. Yifan Yang, Jiajun Zhou, Ngai Wong, and Zheng Zhang. 2024a. Loretta: Low-rank economic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan… find
  23. Yifei Yang, Zouying Cao, and Hai Zhao. 2024b. Laco: Large language model pruning via layer collapse. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6401–6417. find
  24. Hao Yu and Jianxin Wu. 2023. Compressing transformers: features are low-rank, but weights are not! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11007–11015. find
  25. Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2023. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821. — [arXiv:2312.05821](https://arxiv.org/abs/2312.05821) find
  26. Jialin Zhao, Yingtao Zhang, and Carlo Vittorio Cannistraci. 2025. Pivoting factorization: A compact meta low-rank representation of sparsity for efficient inference in large language models. In Forty-second International Conference on Machine Learning. find
  27. Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. 2024. Blockpruner: Fine-grained pruning for large language models. arXiv preprint arXiv:2406.10594. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2026-flat-llm-fine-grained-low/