Elamur Research⌕
☰
home / topics / Efficient models & compression / page 2

🪶 Efficient models & compression

60 papers · page 2 of 2 · 26–60 · the most cited first · all topics

Smaller and faster models: quantization, pruning, distillation and sparse computation.

1 2

  1. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMsSuyu Ge, Yunan Zhang, Liyuan Liu, et al.2023 · 198 citations
  2. Structured Pruning of Large Language ModelsZiheng Wang, Jeremy Wohlwend, Tao Lei2020 · 196 citations
  3. OmniQuant: Omnidirectionally Calibrated Quantization for Large Language ModelsWenqi Shao, Mengzhao Chen, Zhaoyang Zhang, et al.2023 · 193 citations
  4. SqueezeLLM: Dense-and-Sparse QuantizationSehoon Kim, Coleman Hooper, Amir Gholami, et al.2023 · 168 citations
  5. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt CompressionZhuoshi Pan, Qianhui Wu, Huiqiang Jiang, et al.2024 · 159 citations
  6. LayerSkip: Enabling Early Exit Inference and Self-Speculative DecodingMostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, et al.2024 · 155 citations
  7. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information FunnelingZefan Cai, Yichi Zhang, Bofei Gao, et al.2024 · 151 citations
  8. Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative DecodingHeming Xia, Zhe Yang, Qingxiu Dong, et al.2024 · 145 citations
  9. SCOTT: Self-Consistent Chain-of-Thought DistillationPeifeng Wang, Zhengyang Wang, Zheng Li, et al.2023 · 145 citations
  10. MiniLLM: On-Policy Distillation of Large Language ModelsYuxian Gu, Li Dong, Furu Wei, et al.2023 · 143 citations
  11. Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-StepLiunian Harold Li, Jack Hessel, Youngjae Yu, et al.2023 · 139 citations
  12. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse AttentionJingyang Yuan, Huazuo Gao, Damai Dai, et al.2025 · 135 citations
  13. The case for 4-bit precision: k-bit Inference Scaling LawsTim Dettmers, Luke Zettlemoyer2022 · 119 citations
  14. I-BERT: Integer-only BERT QuantizationSehoon Kim, Amir Gholami, Zhewei Yao, et al.2021 · 118 citations
  15. DeepSeek-V4: Towards Highly Efficient Million-Token Context IntelligenceDeepSeek-AI, Anyi Xu, Bangcai Lin, et al.2026 · 117 citations
  16. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice CodebooksAlbert Tseng, Jerry Chee, Qingyao Sun, et al.2024 · 116 citations
  17. DistiLLM: Towards Streamlined Distillation for Large Language ModelsJongwoo Ko, Sungnyun Kim, Tianyi Chen, et al.2024 · 111 citations
  18. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming HeadsGuangxuan Xiao, Jiaming Tang, Jingwei Zuo, et al.2024 · 109 citations
  19. TokenSkip: Controllable Chain-of-Thought Compression in LLMsHeming Xia, Chak Tou Leong, Wenjie Wang, et al.2025 · 105 citations
  20. Language Modeling Is CompressionGregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, et al.2023 · 89 citations
  21. Tokenization Is More Than CompressionCraig W Schmidt, Varshini Reddy, Haoran Zhang, et al.2024 · 66 citations
  22. Improving Stance Detection with Multi-Dataset Learning and Knowledge DistillationYingjie Li, Chenye Zhao, Cornelia Caragea2021 · 39 citations
  23. Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMsNicolas Boizard, Kevin El Haddad, Celine Hudelot, et al.2024 · 34 citations
  24. MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding MethodsMara Finkelstein; Subhajit Naskar; Mehdi Mirzazadeh; Apurva Shah;, Markus Freitag2023 · 30 citations
  25. The Quantization Model of Neural ScalingEric J. Michaud, Ziming Liu, Uzay Girit, et al.2023 · 27 citations
  26. Distillation Scaling LawsDan Busbridge, Amitis Shidani, Floris Weers, et al.2025 · 23 citations
  27. Okay, Let's Do This! Modeling Event Coreference with Generated Rationales and Knowledge DistillationAbhijnan Nath, Shadi Manafi Avari, Avyakta Chelle, et al.2024 · 16 citations
  28. Selected-Layer Codec Compression for Compact Speech Translation Models: An IWSLT 2026 English-to-Chinese SubmissionAlonso Palomino2026 · 1 citation
  29. Rationale-Guided Distillation for E-Commerce Relevance Classification: Bridging Large Language Models and Lightweight Cross-EncodersSanjay Agrawal, Faizan Ahemad, Vivek Varadarajan Sembium2025 · 1 citation
  30. Evaluating Small Language Models for English-to-Portuguese Translation: Impact of Model Scale and QuantizationGustavo Lopes Tamiosso, Rafael Oleques Nunes, Dennis Giovani Balreira2026
  31. Reason-to-Learn (R2L): Multi-Agent Knowledge Distillation for Lightweight LLMs in Sentiment AnalysisLe-Huy Tu, Quan Nguyen, Vincent NGUYEN, et al.2026
  32. SPQ: An Ensemble Technique for Large Language Model CompressionJiamin Yao, Eren Gultepe2026
  33. Terminology-Aware Retrieval-Augmented Knowledge Distillation for Biomedical Neural Machine TranslationMaria Zafar, Souhail Bakkali, Rejwanul Haque2026
  34. 大语言模型推理中的键值缓存压缩方法综述 (Survey on KV Cache Compression Methods for Large Language Model Inference)Shipeng Hu (胡世鹏), Guangyan Zhang (张广艳), Weimin Zheng (郑纬民)2026
  35. 大语言模型推理中的存储优化技术综述 (Survey of Storage Optimization Techniques in Large Language Model Inference)Xuran Ge (葛旭冉), Yang Ou (欧洋), Bo Wang (王博), et al.2025

1 2

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/efficient-models/page/2/