Elamur Research⌕
☰
home / papers / 2026 / Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction

Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction

Paper · 2026 · Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen et al.

Summary

Long-context LLMs are costly to run. KV-cache eviction methods such as H2O and SnapKV make decoding cheaper, but they leave the prefilling phase untouched, and that phase becomes the main bottleneck as inputs grow. The authors observe that attention in early-to-middle layers already points to the tokens that matter for a query. They call these "filter layers" and build GemFilter on this observation. GemFilter runs only the first few layers over the full prompt. It then picks the top-k tokens using the attention scores of the final query position, and feeds this short subset through the complete model for ordinary generation. The method requires no training, works with different LLMs, and lets users read the selected tokens directly. In the experiments on the Needle in a Haystack benchmark, GemFilter outperforms SnapKV as well as regular full attention, and on LongBench it performs comparably to SnapKV/H2O, all while requiring less time and memory. This matters because it reduces cost in both inference phases, not just decoding.

Key points

  • Efficiency: Compared with SnapKV on LLaMA-3.1-8B-Instruct, GemFilter is about 2.4× faster and uses roughly 30% less GPU memory. In decoding, its cost matches SnapKV/H2O.
  • Compression example: On a Needle in a Haystack prompt, the 13th layer of LLaMA-3.1-8B-Instruct served as the filter. It shrank 108,172 tokens to about 100, a ~1000× reduction. The instruction, the needle, and the question all survived the selection. Any of layers 13–19 could serve as the filter.
  • LongBench results: Models tested were LLaMA-3.1-8B-Instruct, Mistral-Nemo-12B-Instruct and Phi-3.5-Mini-3.8B-Instruct. Average scores at a 1024-token budget:
    • Mistral: GemFilter 45.14 vs SnapKV 44.71
    • LLaMA: GemFilter 34.50 vs SnapKV 35.25
    • Phi: GemFilter 32.74 vs SnapKV 33.68
  • Comparison with other compression: LLMLingua-1024 scored much lower than GemFilter. An earlier context-pruning method needed about 50% of tokens, whereas GemFilter keeps roughly 1%.
  • Ablations:
    • Selecting tokens with middle attention rows, or with the rows of largest ℓ2 norm, fails on the needle task. The last query row is essential.
    • Across layers, the model's processing falls into three stages: searching, locating the needle, and preparing the answer.
    • The best filter layer stays stable across inputs.
    • Code is available at github.com/SalesforceAIResearch/GemFilter.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢SalesforceSan Francisco · United States59 works · 18,172 citations🎓University of Hong KongPok Fu Lam · China118 works · 13,496 citations🎓University of Wisconsin–MadisonMadison · United States54 works · 49,089 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang and Shafiq Joty. 2026. Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.677

@inproceedings{shi2026discovering,
  title = {{Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction}},
  author = {Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang and Shafiq Joty},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.677},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.677}
}

References

29 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  2. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. https://www-cdn.anthropic.com. ↗
  3. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.
  4. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055.
  5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, and 1 others. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  6. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.
  7. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. NeurIPS, 35:16344–16359. find
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  9. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801.
  10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  11. Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, and 1 others. 2024a. Minference 1.0: Accelerating prefilling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490.
  12. Huiqiang Jiang, Qianhui Wu, , Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024b. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume…
  13. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023b. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376, Singapore. Association for Comp…
  14. Ziyan Jiang, Xueguang Ma, and Wenhu Chen. 2024c. Longrag: Enhancing retrieval-augmented generation with long-context llms. arXiv preprint arXiv:2406.15319. — [arXiv:2406.15319](https://arxiv.org/abs/2406.15319) find
  15. Greg Kamradt. 2024. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/ LLMTest_NeedleInAHaystack. ↗
  16. Jingyao Li, Han Shi, Xin Jiang, Zhenguo Li, Hong Xu, and Jiaya Jia. 2024a. Quickllama: Query-aware inference acceleration for large language models. arXiv preprint arXiv:2406.07528. find
  17. Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. 2023. Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201.
  18. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024b. Snapkv: Llm knows what you are looking for before generation. NeurIPS.
  19. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Associat…
  20. John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, and 1 others. 2022. Chatgpt: Optimizing language models for dialogue. OpenAI blog, 2(4). find
  21. Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. arXiv preprint arXiv:2407.08608.
  22. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NeurIPS, 30.
  24. Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. 2024. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. arXiv preprint arXiv:2406.18139. find
  25. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, and 1 others. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  26. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. find
  27. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024a. Retrieval meets long context large language models. Preprint, arXiv:2310.03025.
  28. Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. 2024b. Think: Thinner key cache by query-driven pruning. arXiv preprint arXiv:2407.21018. find
  29. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, and 1 others. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. NeurIPS, 36. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-discovering-gems-early-layers-accelerating/