Elamur Research⌕
☰
home / papers / 2026 / Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression

Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression

Paper · 2026 · Luoyang Sun, Guangyan Li, Cheng Deng et al.

Summary

Running large language models is costly. Existing compression tools usually address only one side of the problem. Weight pruning (SparseGPT, WANDA) produces fixed masks, while activation sparsity (TEAL) skips small inputs uniformly. Combining the two has so far required extra machinery, such as the SVD components in R-Sparse. The authors propose DAWS, a training-free method that merges both.

An analysis of pretrained Llama-3-8B shows two things. About 15% of neurons carry roughly 75% of output energy, and sparsity behaviour differs strongly between projection matrices. Each linear layer therefore splits its input by magnitude into three groups:

  • the largest values are multiplied with the original dense weights;
  • mid-range values use a magnitude-pruned copy of the weights;
  • the smallest values are dropped.

A two-stage greedy search sets sparsity budgets across and within modules (q/k/v/o_proj, gate/up/down_proj). The method needs no retraining, decompositions or learned routers.

On Llama-3-8B, Llama-2-7B and Mistral-7B at 50% sparsity, DAWS reportedly keeps over 98% of dense performance and beats WANDA, TEAL and R-Sparse by 1–3 points. This offers a simple path to cheaper inference.

Key points

  • Ablation (Llama-3-8B, 50% sparsity): Pure activation sparsity reaches PPL 8.53 and average accuracy 64.85. The full search (S1 + S2) improves this to PPL 7.17 and 67.83. The dense model scores PPL 6.13 and 69.46. The accuracy average spans eight tasks: Arc-C and Arc-E, BoolQ, HS, OBQA, SciQ, PIQA, and WG.
  • Magnitude vs WANDA-style pruning inside DAWS: Magnitude pruning wins on PPL (7.17 vs 7.30) and on accuracy (67.83 vs 67.18). Masks were compared across WikiText-2 sample pairs. WANDA masks reach IoU of only 0.80–0.90 at best and are least stable in self_attn.o_proj. Magnitude masks do not depend on the data.
  • Dynamic vs static activation masks: Activation masks overlap little between samples (IoU about 0.20–0.35). Replacing them with fixed masks drops average accuracy from 68.15 to 52.36, so thresholds must be computed per input.
  • Theory: The authors give a proof that, under heavy-tailed activations, the dual scheme has lower approximation error than either sparsity type alone.
  • Caveats for readers: The extracted text has internal inconsistencies:
    • DAWS on Llama-3-8B is reported as PPL 6.55 / 68.15 in one table but 7.17 / 67.83 in others. The latter is about 97.7% of dense, below the ">98%" claim.
    • The method overview mentions WANDA-style scoring, while the rest of the paper uses magnitude pruning.
    • Weight-mask IoU figures vary between >0.92 and 0.80–0.90.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓University College LondonLondon · United Kingdom88 works · 19,432 citations🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🔬Institute of AutomationBeijing · China70 works · 14,251 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🎓Guangzhou UniversityGuangzhou · China4 works · 7 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Luoyang Sun, Guangyan Li, Cheng Deng, Haifeng Zhang, Jian Zhao, Yongqiang Tang, Wensheng Zhang and Jun Wang. 2026. Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.378

@inproceedings{sun2026dual,
  title = {{Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression}},
  author = {Luoyang Sun, Guangyan Li, Cheng Deng, Haifeng Zhang, Jian Zhao, Yongqiang Tang, Wensheng Zhang and Jun Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.378},
  url = {https://doi.org/10.18653/v1/2026.acl-long.378}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tai An, Ruwu Cai, Yanzhe Zhang, Yang Liu, Hao Chen, Pengcheng Xie, Sheng Chang, Yiwu Yao, and Gongyi Wang. 2025. Amber pruner: Leveraging n:m activation sparsity for efficient prefill in large language models. Preprint, arXiv:2508.02128. — [arXiv:2508.02128](https://arxiv.org/abs/2508.02128) find
  2. Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proc. AAAI, page 7432–7439.
  3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proc. NAACL.
  4. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  5. Harry Dong, Beidi Chen, and Yuejie Chi. 2024. Prompt-prompted mixture of experts for efficient llm generation. CoRR. find
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  7. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. Preprint, arXiv:1803.03635. find
  8. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pages 10323–10337. PMLR. find
  9. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  10. Song Han, Huizi Mao, and William J. Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. Preprint, arXiv:1510.00149. find
  11. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  12. Donghyun Lee, Je-Yong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. 2024. Cats: Contextually-aware thresholding for sparsity in large language models. arXiv preprint arXiv:2404.08763. find
  13. James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. 2024. Training-free activation sparsity in large language models. arXiv preprint arXiv:2408.14690. find
  14. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Preprint, arXiv:2305.11627.
  15. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
  16. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proc. EMNLP. find
  17. Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. 2023. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04564. — [arXiv:2310.04564](https://arxiv.org/abs/2310.04564) find
  18. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proc. AAAI, page 8732–8740. find
  19. Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. Powerinfer: Fast large language model serving with a consumer-grade gpu. Preprint, arXiv:2312.12456. find
  20. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695. find
  21. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  22. Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106.
  23. Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, and 1 others. 2025. Wanda++: Pruning large language models via regional gradients. arXiv preprint arXiv:2503.04992. find
  24. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proc. ACL. find
  25. Zhenyu Zhang, Zechun Liu, Yuandong Tian, Harshit Khaitan, Zhangyang Wang, and Steven Li. 2025. R-sparse: Rank-aware activation sparsity for efficient llm inference. In The Thirteenth International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-dual-activation-weight-sparsity-training/