Elamur Research⌕
☰
home / papers / 2026 / ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks

ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks

Paper · 2026 · Zhixiong Zhuang, Maria-Irina Nicolae, Hui-Po Wang et al.

Summary

System prompts for LLM applications often represent significant engineering effort and may contain confidential rules, filtering criteria or business logic. Attackers can nonetheless trick models into disclosing them. Existing countermeasures have clear weaknesses. Instructing the model to keep the prompt secret can be overridden by adversarial queries. Filtering outputs for overlap with the prompt fails against translation or encoding tricks.

ProxyPrompt takes a different approach. A defender with access to model weights optimizes a substitute prompt directly in embedding space using a joint objective:

  • one term keeps the model's answers to representative task queries close to those produced with the original prompt;
  • the other term pushes whatever the model reveals under a simple extraction query away from the original text.

Because the optimized embeddings lie far from real vocabulary tokens, converting them back into words loses additional information. Across 264 model–prompt configurations, tested against a pooled set of published attacks, the method protected 94.70% of prompts. The strongest baseline protected 42.80%. This gives application owners a practical way to guard sensitive instructions without relying on the model's willingness to obey.

Key points

  • Headline result: 94.70% of prompts protected versus 42.80% for the second-best defense (Filter).
  • Evaluation setup:
    • Tasks span reasoning, role-playing and classification (GSM8K, CoLA, SST-2, QNLI, Roles).
    • The attack set combines queries from four prior works, including Raccoon and Pleak.
    • Additional tests cover 834-token CoT prompts and ALFWorld.
  • Why extraction fails: Proxy embeddings have a cosine similarity of only about 0.11–0.12 to their nearest tokens (victim model L-8B). Utility stays high even with just N = 5 relevant queries.
  • HuggingChat case study: On the "Image Generator" assistant with an added secret commercial rule, using L-70B, utility (UR) reached 1.00 and leakage metrics (SM, AM) were 0.
  • Extensions, metric and limits:
    • Proxies can be concatenated with non-sensitive instructions without re-optimization.
    • The authors propose a semantic-level leakage metric.
    • The method requires access to model internals, so it does not apply directly to closed API models.
    • Code: github.com/boschresearch/proxyprompt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations🔬Helmholtz Center for Information SecuritySaarbrücken · Germany14 works · 1,172 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

Cite

Cite as Zhixiong Zhuang, Maria-Irina Nicolae, Hui-Po Wang and Mario Fritz. 2026. ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.429

@inproceedings{zhuang2026proxyprompt,
  title = {{ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks}},
  author = {Zhixiong Zhuang, Maria-Irina Nicolae, Hui-Po Wang and Mario Fritz},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.429},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.429}
}

References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners…
  2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  3. Hai Dang, Lukas Mecke, Florian Lehmann, Sven Goller, and Daniel Buschek. 2022. How to prompt? opportunities and challenges of zero-and few-shot learning for human-ai interaction in creative applications of generative models. In Generative AI and HCI Workshop. find
  4. Edoardo Debenedetti, Javier Rando, Daniel Paleka, Silaghi Fineas Florin, Dragos Albastroiu, Niv Cohen, Yuval Lemberg, Reshmi Ghosh, Rui Wen, Ahmed Salem, and 1 others. 2024. Dataset and lessons learned from the 2024 satml llm capture-the-flag competition. arXiv preprint arXiv:2406.07954. — [arXiv:2406.07954](https://arxiv.org/abs/2406.07954) find
  5. Deepseek-Team. 2023. Deepseek-llm-7b-chat. https://huggingface.co/deepseek-ai/ deepseek-llm-7b-chat. Accessed: 2025-2-20. ↗
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations (ICLR).
  8. Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. 2024. Defending against indirect prompt injection attacks with spotlighting. arXiv preprint arXiv:2403.14720. — [arXiv:2403.14720](https://arxiv.org/abs/2403.14720) find
  9. HuggingChat. 2024. Huggingchat assistants. https://huggingface.co/chat/assistants. Accessed: 2025-1-18. ↗
  10. Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, and Yinzhi Cao. 2024. Pleak: Prompt leaking attacks against large language model applications. In The ACM Conference on Computer and Communications Security (CCS). find
  11. Wynter Jones. 2023. chatgpt-roles. https://huggingface.co/datasets/WynterJones/ chatgpt-roles. Accessed: 2025-1-18. ↗
  12. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems (NeurIPS).
  13. Donggyu Lee. 2023. leaked system prompts. https://github.com/jujumilk3/ leaked-system-prompts. Accessed: 2025-1-18. ↗
  14. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  15. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP). find
  16. Zi Liang, Haibo Hu, Qingqing Ye, Yaxin Xiao, and Haoyang Li. 2024. Why are my prompts leaked? unraveling prompt extraction threats in customized large language models. arXiv preprint arXiv:2408.02416. — [arXiv:2408.02416](https://arxiv.org/abs/2408.02416) find
  17. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR). find
  18. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft. Accessed: 2025-1-18. ↗
  19. Bertalan Meskó. 2023. Prompt engineering as an important emerging skill for medical professionals: tutorial. Journal of medical Internet research. find
  20. Meta. 2025. Llama prompt guard 2. https://www.llama.com/docs/ model-cards-and-prompt-formats/ prompt-guard/. ↗
  21. Mistral-Team. 2023. Mistral-7b-instruct-v0.2. https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.2t. Accessed: 2025-2-20. ↗
  22. OpenAI. 2023. Gpt-3.5 turbo fine-tuning and api updates. https://openai.com/blog/ gpt-3-5-turbo-fine-tuning-and-api-updates. Accessed: 2025-1-18. ↗
  23. OpenAI. 2024. Gpt store. https://openai.com/ index/introducing-the-gpt-store/. Accessed: 2025-1-18. ↗
  24. OWASP. 2025. Prompt leakage threat. https://genai.owasp.org/llmrisk/ llm072025-system-prompt-leakage/. Accessed: 2025-1-18. ↗
  25. David Pape, Sina Mavali, Thorsten Eisenhofer, and Lea Schönherr. 2025. Prompt obfuscation for large language models. In USENIX Security. find
  26. Poe. 2024. Poe bot. https://poe.com/. Accessed: 2025-1-18. ↗
  27. Qwen-Team. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  29. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  30. Louis Shark. 2023. Promptcraft: The ultimate gpt system prompt collection. https://github.com/ LouisShark/chatgpt_system_prompt. Accessed: 2025-1-18. ↗
  31. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations (ICLR).
  32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  33. Victor. 2024. Image generator. https://hf.co/chat/ Ac-assistant/65bff23f5560c1a5c0c9dcbd. cessed: 2025-5-10. ↗
  34. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR).
  35. Junlin Wang, Tianyi Yang, Roy Xie, and Bhuwan Dhingra. 2024a. Raccoon: Prompt extraction benchmark of llm-integrated applications. In Findings of the Association for Computational Linguistics (ACL). find
  36. Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P Xing, and Zhiting Hu. 2024b. Promptagent: Strategic planning with language models enables expert-level prompt optimization. In International Conference on Learning Representations (ICLR). find
  37. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471. find
  38. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhuang-2026-proxyprompt-securing-system-prompts-against/