Elamur Research⌕
☰
home / papers / 2026 / CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editing

CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editing

Paper · 2026 · Rui Wang, Junda Wu, Yu Xia et al.

Summary

The paper addresses indirect prompt injection, where an LLM obeys commands hidden inside supplied context (e.g., a retrieved passage) instead of answering the user's actual request. The authors attribute this to the model failing to separate data from instructions. Their method, CachePrune, works in two stages. First, it finds neurons in the key/value activations of the context that push the model toward obeying injected text. It does this with a gradient-times-activation attribution score driven by a "preferential attribution loss," which compares sampled clean and poisoned responses and is shown to bound the DPO objective from above. Second, it masks those neurons only when building the KV cache for the context span. A regularizing subset keeps user-instruction following intact. The attribution also exploits a "triggering effect": the first few response tokens best reveal whether the output will be poisoned. Because only a pruning mask is applied, CachePrune leaves the prompt format untouched and adds no generation-time cost. It also fits context-caching setups. Experiments, mainly on SQuAD with Llama3-8B, show large drops in attack success rate (ASR) while clean-answer F1 is preserved.

Key points

  • Attribution needs only a few prompt samples. Each neuron's scores are aggregated by taking the maximum over context tokens, and up to p% of neurons are pruned from a "safe" subset Φ.
  • Using only the earliest response tokens works better than using the full response. On SQuAD with Llama3-8B, ASR is 5.57 with k=2 and 7.44 with k=1, versus 14.81 when all tokens are used.
  • Masks transfer across injection types:
    • Code injections: ASR drops from 17.5 (no defense) to 1.77 with a code-learned mask and 3.20 with a text-learned mask.
    • Text injections: ASR drops from 45.15 to 9.23 (text-learned mask) and 16.9 (code-learned mask).
    • Sources: code tasks come from Chaudhary (2023) and text tasks from Ji et al. (2023).
  • Masks also generalize across four different attack instructions, with off-diagonal transfers performing comparably to matched ones.
  • Pruned neurons cluster in middle layers and are mostly key rather than value neurons. This suggests the method changes attention to the context while largely preserving the context's content.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California San DiegoSan Diego · United States73 works · 51,948 citations🏢Adobe SystemsSan Jose · United States35 works · 3,711 citations🎓UNSW SydneySydney · Australia34 works · 1,509 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Rui Wang, Junda Wu, Yu Xia, Tong Yu, Ruiyi Zhang, Ryan A. Rossi, Subrata Mitra, Lina Yao and Julian McAuley. 2026. CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editing. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.70

@inproceedings{wang2026cacheprune,
  title = {{CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editing}},
  author = {Rui Wang, Junda Wu, Yu Xia, Tong Yu, Ruiyi Zhang, Ryan A. Rossi, Subrata Mitra, Lina Yao and Julian McAuley},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.70},
  url = {https://doi.org/10.18653/v1/2026.acl-long.70}
}

References

48 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2025. gemin-cc. http://ai.google.dev/ gemini-api/docs/caching?lang=python. 2025. openai-cc. https://openai.com/index/ api-prompt-caching/. 2025. Owasp. https://genai.owasp.org/. ↗
  2. Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, and Andrew Paverd. 2024. Are you still on track!? catching llm task drift with activations. arXiv preprint arXiv:2406.00799. — [arXiv:2406.00799](https://arxiv.org/abs/2406.00799) find
  3. Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, and Andrew Paverd. 2025. Get my drift? catching llm task drift with activation deltas. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 43–67. IEEE. find
  4. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. Phi-… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  5. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  6. Lin Ai, Zheng Hui, Zizhou Liu, and Julia Hirschberg. 2024. Enhancing pre-trained generative language models with question attended span extraction on machine reading comprehension. arXiv preprint arXiv:2404.17991. find
  7. Md Ahsan Ayub and Subhabrata Majumdar. 2024. Embedding-based classifiers can detect prompt injection attacks. arXiv preprint arXiv:2410.22284. — [arXiv:2410.22284](https://arxiv.org/abs/2410.22284) find
  8. Jonas Becker, Jan Philip Wahle, Bela Gipp, and Terry Ruas. 2024. Text generation: A systematic literature review of tasks, evaluation, and challenges. arXiv preprint arXiv:2405.15604. — [arXiv:2405.15604](https://arxiv.org/abs/2405.15604) find
  9. Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca. ↗
  10. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2024a. Struq: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363. — [arXiv:2402.06363](https://arxiv.org/abs/2402.06363) find
  11. Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, and Chuan Guo. 2024b. Aligning llms to be robust against prompt injection. arXiv preprint arXiv:2410.05451. — [arXiv:2410.05451](https://arxiv.org/abs/2410.05451) find
  12. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495. find
  13. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security,…
  14. Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. 2024. Defending against indirect prompt injection attacks with spotlighting. arXiv preprint arXiv:2403.14720. — [arXiv:2403.14720](https://arxiv.org/abs/2403.14720) find
  15. Chengkai Huang, Kaige Xie, Rui Wang, Tong Yu, and Lina Yao. 2024. Learn when (not) to trust language models: A privacy-centric adaptive model-aware approach. arXiv preprint arXiv:2404.03514. — [arXiv:2404.03514](https://arxiv.org/abs/2404.03514) find
  16. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704.
  17. Feiran Jia, Tong Wu, Xin Qin, and Anna Squicciarini. 2025. The task shield: Enforcing task alignment to defend against indirect prompt injection in llm agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29680–29697.
  18. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  19. Hao Li, Xiaogeng Liu, Ning Zhang, and Chaowei Xiao. 2025. Piguard: Prompt injection guardrail via mitigating overdefense for free. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 30420–30437.
  20. Miles Q Li and Benjamin CM Fung. 2025. Security concerns for large language models: A survey. Journal of Information Security and Applications, 95:104284.
  21. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  22. Ruofan Liu, Yun Lin, Zhiyong Huang, and Jin Song Dong. 2025. Drip: Defending prompt injection via token-wise representation editing and residual instruction fusion. arXiv preprint arXiv:2511.00447. — [arXiv:2511.00447](https://arxiv.org/abs/2511.00447) find
  23. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024. Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957. — [arXiv:2403.04957](https://arxiv.org/abs/2403.04957) find
  24. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2023. Prompt injection attacks and defenses in llm-integrated applications. arXiv preprint arXiv:2310.12815. find
  25. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527.
  26. Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. 2024. Jatmo: Prompt injection defense by task-specific finetuning. In European Symposium on Research in Computer Security, pages 105–124. Springer Nature Switzerland Cham. find
  27. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.
  28. P Rajpurkar. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
  29. Sander Schulhoff, Jeremy Pinto, Anaum Khan, L-F Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, Christopher Carnahan, and Jordan Boyd-Graber. 2023. Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global scale prompt hacking competition.… find
  30. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In International conference on machine learning, pages 3145–3153. PMlR.
  31. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  32. Prashant Upadhyay, Rishabh Agarwal, Sumeet Dhiman, Abhinav Sarkar, and Saumya Chaturvedi. 2024. A comprehensive survey on answer generation methods using nlp. Natural Language Processing Journal, 8:100088. find
  33. Jiongxiao Wang, Fangzhou Wu, Wendi Li, Jinsheng Pan, Edward Suh, Z Morley Mao, Muhao Chen, and Chaowei Xiao. 2024. Fath: Authentication-based test-time defense against indirect prompt injection attacks. arXiv preprint arXiv:2410.21492. — [arXiv:2410.21492](https://arxiv.org/abs/2410.21492) find
  34. A Waswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, A Gomez, L Kaiser, and I Polosukhin. 2017. Attention is all you need. In NIPS.
  35. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079–80110.
  36. Fangzhao Wu, Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, and Xing Xie. 2023. Defending chatgpt against jailbreak attack via self-reminder. find
  37. Fangzhou Wu, Shutong Wu, Yulong Cao, and Chaowei Xiao. 2024. Wipi: A new web threat for llm-driven web agents. arXiv preprint arXiv:2402.16965. — [arXiv:2402.16965](https://arxiv.org/abs/2402.16965) find
  38. Nakyeong Yang, Yunah Jang, Hwanhee Lee, Seohyeong Jung, and Kyomin Jung. 2022. Task-specific compression for multi-task language models using attribution-based pruning. arXiv preprint arXiv:2205.04157. find
  39. Nakyeong Yang, Taegwan Kang, Jungkyu Choi, Honglak Lee, and Kyomin Jung. 2023. Mitigating biases for instruction-following language models via bias neurons elimination. arXiv preprint arXiv:2311.09627. find
  40. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.
  41. Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2023. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv preprint arXiv:2312.14197. — [arXiv:2312.14197](https://arxiv.org/abs/2312.14197) find
  42. Jiahao Yu, Yuhang Wu, Dong Shu, Mingyu Jin, and Xinyu Xing. 2023. Assessing prompt injection risks in 200+ custom gpts. arXiv preprint arXiv:2311.11538. — [arXiv:2311.11538](https://arxiv.org/abs/2311.11538) find
  43. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv preprint arXiv:2403.02691.
  44. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In Proceedings of the International Conference on Learning Representations (ICLR). find
  45. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470.
  46. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.
  47. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  48. Egor Zverev, Sahar Abdelnabi, Soroush Tabesh, Mario Fritz, and Christoph H Lampert. 2024. Can llms separate instructions from data? and what do we even mean by that? arXiv preprint arXiv:2403.06833. — [arXiv:2403.06833](https://arxiv.org/abs/2403.06833) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-cacheprune-teaching-llms-what-not/