Elamur Research⌕
☰
home / papers / 2026 / PARASITE: Conditional System Prompt Poisoning to Hijack LLMs

PARASITE: Conditional System Prompt Poisoning to Hijack LLMs

Paper · 2026 · Viet Pham and Thai Le

Summary

The paper examines a supply-chain risk in which people download ready-made system prompts from public marketplaces such as FlowGPT and use them without checking. The authors define Conditional System Prompt Poisoning: a prompt that looks harmless and works well on ordinary questions, but pushes the model toward a chosen wrong answer when one specific question is asked. Their framework, PARASITE, builds such prompts using only black-box API access. It works in two stages. First, a global semantic search produces a readable prompt that is already partly adversarial. Second, greedy word-level edits, including small typo-like changes, strengthen the attack while keeping benign accuracy. Experiments use TriviaQA and TruthfulQA questions on open-source models and commercial APIs. PARASITE cuts F1 on the targeted queries by up to 70% and costs about $2 per target, with little loss on other queries. Standard filters fail to stop it because real-world prompts already contain natural errors. The work shows that prompt marketplaces need defenses that test how a prompt behaves, not just how it reads.

Key points

  • The attack was tested on GPT-4o-mini, GPT-3.5-Turbo, Llama-3, Qwen-2.5 and DeepSeek-7B, in TwoOptions, FourOptions and Freeform answer formats.
  • The optimization balances two goals: an adversarial loss on the target set and a utility loss on benign questions. Low perplexity and low similarity to the target question are imposed as stealth constraints.
  • Rewriting the prompt with GPT-4o-mini does not remove the attack:
    • With polishing, DeepSeek TwoOptions malicious F1 only falls from 58.9 to 55.8.
    • With strong paraphrasing, Qwen2.5-7B TwoOptions malicious F1 stays at 64.3.
  • Sanitization also damages legitimate performance: Qwen2.5-7B Freeform benign F1 drops from 41.6 to about 22.
  • Paraphrasing the user's question does not help either, since malicious F1 stays above 60%. The study is limited to single-turn, English-language interactions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indiana University BloomingtonBloomington · United States37 works · 3,685 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Viet Pham and Thai Le. 2026. PARASITE: Conditional System Prompt Poisoning to Hijack LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.668

@inproceedings{pham2026parasite,
  title = {{PARASITE: Conditional System Prompt Poisoning to Hijack LLMs}},
  author = {Viet Pham and Thai Le},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.668},
  url = {https://doi.org/10.18653/v1/2026.acl-long.668}
}

References

38 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson E. Denison, Evan Hubinger, and 15 othe… find
  2. Azure. 2024. Azure ai content safety. find
  3. Advik Raj Basani and Xiao Zhang. 2024. Gasp: Efficient black-box generation of adversarial suffixes for jailbreaking llms. Preprint, arXiv:2411.14133. — [arXiv:2411.14133](https://arxiv.org/abs/2411.14133) find
  4. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. Preprint, arXiv:2310.08419.
  5. Yangyi Chen, Hongcheng Gao, Ganqu Cui, Fanchao Qi, Longtao Huang, Zhiyuan Liu, and Maosong Sun. 2022. Why should adversarial perturbations be imperceptible? rethink the research paradigm in adversarial nlp. arXiv preprint arXiv:2210.10683. find
  6. DeepSeek-AI. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  7. Martin Dietzfelbinger, Andreas Goerdt, Michael Mitzenmacher, Andrea Montanari, Rasmus Pagh, and Michael Rink. 2010. Tight thresholds for cuckoo hashing via xorsat. Preprint, arXiv:0912.0287. — [arXiv:0912.0287](https://arxiv.org/abs/0912.0287) find
  8. J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50– 56.
  9. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security.
  10. Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. Cold-attack: Jailbreaking llms with stealthiness and controllability. arXiv preprint arXiv:2402.08679. — [arXiv:2402.08679](https://arxiv.org/abs/2402.08679) find
  11. Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tramèr, and Milad Nasr. 2024. Query-based adversarial prompt generation. ArXiv, abs/2402.12329. — [arXiv:2402.12329](https://arxiv.org/abs/2402.12329) find
  12. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  13. Evan Hubinger, Carson E. Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte Stuart MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Kristjanson Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, and 20… find
  14. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. Preprint, arXiv:2312.06674.
  15. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. Preprint, arXiv:2309.00614.
  16. Slobodan Jenko, Niels Mündler, Jingxuan He, Mark Vero, and Martin Vechev. 2025. Black-box adversarial attacks on llm-based code completion. Preprint, arXiv:2408.02509. — [arXiv:2408.02509](https://arxiv.org/abs/2408.02509) find
  17. Weipeng Jiang, Zhenting Wang, Juan Zhai, Shiqing Ma, Zhengyu Zhao, and Chao Shen. 2025. An optimizable suffix is worth a thousand templates: Efficient black-box jailbreaking without affirmative phrases via llm as optimizer. Preprint, arXiv:2408.11313. find
  18. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is bert really robust? natural language attack on text classification and entailment. arXiv preprint arXiv:1907.11932.
  19. Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.
  20. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver…
  21. Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during instruction tuning. In International Conference on Machine Learning. find
  22. Elad Levi, Eli Brosh, and Matan Friedmann. 2024. Intent-based prompt calibration: Enhancing prompt optimization with synthetic boundary cases. Preprint, arXiv:2402.03099. — [arXiv:2402.03099](https://arxiv.org/abs/2402.03099) find
  23. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  24. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yanhong Zheng, and Yang Liu. 2023. Prompt injection attack against llm-integrated applications. ArXiv, abs/2306.05499.
  25. Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan Kankanhalli. 2024. An LLM can fool itself: A prompt-based adversarial attack. In The Twelfth International Conference on Learning Representations. find
  26. Jiaqi Xue, Mengxin Zheng, Ting Hua, Yilin Shen, Yepeng Liu, Ladislau Boloni, and Qian Lou. 2023. Trojllm: A black-box trojan prompt attack on large language models. Preprint, arXiv:2306.06815. — [arXiv:2306.06815](https://arxiv.org/abs/2306.06815) find
  27. Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. 2024. Backdooring instruction-tuned large language models with virtual prompt injection. Preprint, arXiv:2307.16888. find
  28. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers. arXiv preprint arXiv:2309.03409.
  29. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. Preprint, arXiv:2402.04249.
  30. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2024. AutoDAN: Interpretable gradient-based adversarial attacks on large language models. In First Conference on Language Modeling. find
  31. George A. Miller. 1994. WordNet: A lexical database for English. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
  32. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. Preprint, arXiv:2307.15043.
  33. OpenAI. 2024. Gpt-4 technical report. arXiv:2303.08774.
  34. Preprint, OpenAI. 2025. Gpt-5 system card. find
  35. Qwen. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  36. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  37. Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2024. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack. ArXiv, abs/2404.01833. find
  38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pham-2026-parasite-conditional-system-prompt-poisoning/