Elamur Research⌕
☰
home / papers / 2026 / AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse

AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse

Paper · 2026 · Jie Ou, Jinyu Guo, Shiyao Guo et al.

Summary

Many-shot in-context learning (ICL) puts hundreds or thousands of demonstrations into an LLM prompt. Current methods use one fixed shot count for every query. The authors show this is a poor fit: accuracy does not rise steadily with more examples, and the best count depends on the model, the task and the individual query. Long prompts also make prefilling slow and memory-hungry.

AdapShot tackles both problems. At inference time it runs cheap single-token probes at increasing shot counts. It measures output entropy as a confidence signal and stops at the smallest context that falls below a threshold. To avoid recomputing context for every probe, demonstrations are prefilled once offline into a semantics-aware global KV cache pool. A position decoupling and re-encoding step uses rotations based on RoPE properties, so cached blocks can be reordered and joined without new forward passes.

Across several LLMs (Qwen2.5, Llama-3.x, openPangu) and benchmarks, AdapShot beats the state-of-the-art DBSA by about 10% on average and runs 4.64× faster. This makes many-shot ICL more practical to deploy.

Key points

  • Motivation: The shot count and accuracy are not monotonically related.
    • Llama-3.1-8B on OpenBookQA drops from 76% (256 shots) to 73% (512 shots), and runs out of memory at 1024.
    • Qwen2.5-7B on QNLI peaks at 78% with 256 shots.
  • Probing: Probes are evaluated in parallel batches (e.g., 4/8/12/16 shots, then 20–32) against an entropy threshold (0.65 in the case study). In a SQuAD example with Qwen2.5-7B, the method chose 28 shots and answered correctly. The authors report the method is not sensitive to the exact threshold τ.
  • BM25 comparison: With adaptive shot counts, AdapShot outperforms fixed BM25-retrieved 256/512-shot prompts.
    • GSM8K: 0.649 vs. 0.456
    • PIQA: 0.599 vs. 0.388
    • SQuAD v2: 0.465 vs. 0.418
  • Scaling: On Qwen2.5-32B (CoLA), AdapShot reaches 86.41% Exact Match with only about 60 shots. A fixed 256-shot baseline falls to 67.91%, suggesting that larger models are more sensitive to context noise.
  • openPangu-Embedded-1B-V1.1: AdapShot speeds up inference in every setting tested, up to 6.73× versus 128-shot on GSM8K. Its main limitation is that it still depends on curated real demonstrations; the authors propose synthetic "implicit shots" as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Electronic Science and Technology of ChinaChengdu · China31 works · 2,012 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Jie Ou, Jinyu Guo, Shiyao Guo, Yuang Li, Ruiqi Wu, Zhaokun Wang, Wenyi Li and Wenhong Tian. 2026. AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1990

@inproceedings{ou2026adapshot,
  title = {{AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse}},
  author = {Jie Ou, Jinyu Guo, Shiyao Guo, Yuang Li, Ruiqi Wu, Zhaokun Wang, Wenyi Li and Wenhong Tian},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1990},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1990}
}

References

37 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shantanu Acharya, Fei Jia, and Boris Ginsburg. 2024. Star attention: Efficient llm inference over long sequences. In Forty-second International Conference on Machine Learning. find
  2. Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, and 1 others. 2024. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37:76930– 76966.
  3. Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. 2025. In-context learning with long-context models: An in-depth exploration. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lingu…
  4. Rahul Atul Bhope, Praveen Venkateswaran, KR Jayaram, Vatche Isahagian, Vinod Muthusamy, and Nalini Venkatasubramanian. 2025. Optiseq: Ordering examples on-the-fly for in-context learning. arXiv preprint arXiv:2501.15030. find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Sihan Cao, Jianwei Zhang, Pengcheng Zheng, Jiaxin Yan, Caiyan Qin, Yalan Ye, Wei Dong, Peng Wang, Yang Yang, and Chaoning Zhang. 2026. Language-guided token compression with reinforcement learning in large vision-language models. arXiv preprint arXiv:2603.13394. — [arXiv:2603.13394](https://arxiv.org/abs/2603.13394) find
  7. Hanting Chen, Yasheng Wang, Kai Han, Dong Li, Lin Li, Zhenni Bi, Jinpeng Li, Haoyu Wang, Fei Mi, Mingjian Zhu, and 1 others. 2025a. Pangu embedded: An efficient dual-system llm reasoner with metacognition. arXiv preprint arXiv:2505.22375.
  8. Zihan Chen, Song Wang, Zhen Tan, Jundong Li, and Cong Shen. 2025b. Maple: Many-shot adaptive pseudo-labeling for in-context learning. In Forty-second International Conference on Machine Learning. find
  9. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509.
  10. Shahriar Golchin, Yanfei Chen, Rujun Han, Manan Gandhi, Tianli Yu, Swaroop Mishra, Mihai Surdeanu, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. 2025. Towards compute-optimal many-shot in-context learning. In Second Conference on Language Modeling.
  11. Zhengyao Gu, Henry Peng Zou, Aiwei Liu, Yankai Chen, Weizhi Zhang, and Philip S Yu. 2025. Scaling laws for many-shot in-context learning with self-generated annotations. In ICML 2025 Workshop on Long-Context Foundation Models. find
  12. Yinhan He, Wendy Zheng, Song Wang, Zaiyi Zheng, Yushun Dong, Yaochen Zhu, and Jundong Li. 2024. Hierarchical demonstration order optimization for many-shot in-context learning.
  13. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Llmlingua: Compressing prompts for accelerated inference of large language models. Preprint, arXiv:2310.05736.
  14. Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. 2023. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. arXiv preprint arXiv:2305.14045.
  15. Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. 2023. In-context learning with many demonstration examples. arXiv preprint arXiv:2302.04931. — [arXiv:2302.04931](https://arxiv.org/abs/2302.04931) find
  16. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947–22970.
  17. Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024. Minicache: Kv cache compression in depth dimension for large language models. Advances in Neural Information Processing Systems, 37:139997–140031. find
  18. Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Yue Liu, Bo Li, Xuming Hu, and Xiaowen Chu. 2025. Chunkkv: Semantic-preserving kv cache compression for efficient long-context llm inference. arXiv preprint arXiv:2502.00299. find
  19. Saeed Moayedpour, Alejandro Corrochano-Navarro, Faryad Sahneh, Alexander Koetter, Jiří Vymětal, Lorenzo Kogler Anele, Pablo Mas, Yasser Jangjoo, Sizhen Li, Michael Bailey, and 1 others. 2024. Many-shot in-context learning for molecular inverse design. In ICML 2024 AI for Science Workshop. find
  20. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
  21. Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. Parallel context windows for large language models. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1…
  22. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  24. Xingchen Wan, Han Zhou, Ruoxi Sun, and Sercan O Arik. 2025. From few to many: Self-improving many-shot reasoners through iterative optimization and generation. In The Thirteenth International Conference on Learning Representations. find
  25. Haonan Wang, Qian Liu, Chao Du, Tongyao Zhu, Cunxiao Du, Kenji Kawaguchi, and Tianyu Pang. 2024. When precision meets position: Bfloat16 breaks down rope in long-context training. Transactions on Machine Learning Research. find
  26. Emily Xiao, Chin-Jou Li, Yilin Zhang, Graham Neubig, and Amanda Bertsch. 2025. Efficient many-shot in-context learning with dynamic block-sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31946–31958, Vienna, A… find
  27. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations. find
  28. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 14…
  29. Qingyu Yin, Xuzheng He, Chak Tou Leong, Fan Wang, Yanzhao Yan, Xiaoyu Shen, and Qiang Zhang. 2024. Deeper insights without updates: The power of in-context learning over fine-tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4138–4151. find
  30. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and 1 others. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297.
  31. Jiaquan Zhang, Qigan Sun, Chaoning Zhang, Xudong Wang, Zhenzhen Huang, Yitian Zhou, Pengcheng Zheng, Chi-lok Andy Tai, Sung-Ho Bae, Zeyu Ma, and 1 others. 2026a. Tda-rc: Task-driven alignment for knowledge-based reasoning chains in large language models. arXiv preprint arXiv:2604.04942. — [arXiv:2604.04942](https://arxiv.org/abs/2604.04942) find
  32. Jiaquan Zhang, Chaoning Zhang, Shuxu Chen, Xudong Wang, Zhenzhen Huang, Pengcheng Zheng, Shuai Yuan, Sheng Zheng, Qigan Sun, Jie Zou, and 1 others. 2026b. Learning global hypothesis space for enhancing synergistic reasoning chain. arXiv preprint arXiv:2602.09794. — [arXiv:2602.09794](https://arxiv.org/abs/2602.09794) find
  33. Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John Lui, and Haibo Chen. 2024. Unifying kv cache compression for large language models with leankv. arXiv preprint arXiv:2412.03131. find
  34. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, and 1 others. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:346… find
  35. Pengcheng Zheng, Xiaorong Pu, Kecheng Chen, Jiaxin Huang, Meng Yang, Bai Feng, Yazhou Ren, Jianan Jiang, Chaoning Zhang, Yang Yang, and 1 others. 2025. Joint lossless compression and steganography for medical images via large language models. arXiv preprint arXiv:2508.01782. — [arXiv:2508.01782](https://arxiv.org/abs/2508.01782) find
  36. Pengcheng Zheng, Chaoning Zhang, Jiarong Mo, Guo-Hui Li, Jiaquan Zhang, Jiahao Zhang, Sihan Cao, Sheng Zheng, Caiyan Qin, Guoqing Wang, and 1 others. 2026. Llava-fa: Learning fourier approximation for compressing large multimodal models. arXiv preprint arXiv:2602.00135. — [arXiv:2602.00135](https://arxiv.org/abs/2602.00135) find
  37. Kaijian Zou, Muhammad Khalifa, and Lu Wang. 2025. On many-shot in-context learning for long-context evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25605–25639.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ou-2026-adapshot-adaptive-many-shot-context/