Elamur Research⌕
☰
home / papers / 2024 / PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models

PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models

Paper · 2024 · Wei Zou, Runpeng Geng, Binghui Wang et al.

Summary

Retrieval-Augmented Generation (RAG) lets an LLM answer questions using texts pulled from an external knowledge database. Most prior work on RAG has targeted accuracy and efficiency, not security. This paper treats the knowledge database as an attack surface: an adversary who can add content, for example by editing Wikipedia, posting web pages, or acting as an insider, can steer the model's output. The authors present PoisonedRAG, which they describe as the first knowledge corruption attack on RAG. The attacker picks target questions and arbitrary target answers, then plants a few crafted passages. Each passage combines two parts. One part makes it rank highly for the target question (the retrieval condition). The other part pushes the LLM toward the chosen answer (the generation condition). The attack has variants for white-box access to the retriever and for a fully black-box setting. Across benchmarks, retrievers and LLMs, a handful of poisoned texts hidden among millions of clean ones succeeded most of the time. Tested defenses did not stop it, which shows that RAG deployments need new protections.

Key points

  • Evaluation covered NQ, HotpotQA and MS-MARCO, 8 LLMs (including GPT-4 and LLaMA-2), and real-world settings such as advanced RAG schemes, a Wikipedia-based chatbot and an LLM agent. The metric was Attack Success Rate (ASR).
  • On NQ, inserting 5 malicious texts per target question into a database of 2,681,468 clean texts gave 97% ASR in the black-box setting. Five baselines, including Zhong et al.'s adversarial passages, stayed below 70%.
  • Paraphrasing, perplexity-based detection and SHA-256 duplicate filtering all failed as defenses. Duplicate filtering left ASR unchanged because each GPT-4-generated sub-text is unique. With knowledge expansion at k = 50, ASR on HotpotQA was still 41% (black-box) and 43% (white-box).
  • The poisoned texts had little effect on unrelated questions: 0.3% (black-box) and 0.9% (white-box) of non-target questions retrieved them.
  • The attack also worked for fact verification on FEVER, reaching 0.97 (black-box) and 0.88 (white-box) ASR. The code is released on GitHub (sleeepeer/PoisonedRAG).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

 Fields, topics and concepts

Cite

Cite as Wei Zou, Runpeng Geng, Binghui Wang and Jinyuan Jia. 2024. PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models. arXiv:2402.07867.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.07867 · arXiv 2402.07867

@misc{zou2024poisonedrag,
  title = {{PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models}},
  author = {Wei Zou, Runpeng Geng, Binghui Wang and Jinyuan Jia},
  year = {2024},
  doi = {10.48550/arxiv.2402.07867},
  eprint = {2402.07867},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.07867}
}

References

107 references, 46 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” NeurIPS, 2020.
  2. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv, 2023.
  3. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen et al., “Palm 2 technical report,” arXiv, 2023.
  4. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Computing Surveys, vol. 55, no. 12, pp. 1–38, 2023.
  5. Y. Al Ghadban, H. Y. Lu, U. Adavi, A. Sharma, S. Gara, N. Das, B. Kumar, R. John, P. Devarsetty, and J. E. Hirst, “Transforming healthcare education: Harnessing large language models for frontline health worker capacity building using retrieval-augmented generation,” medRxiv, pp. 2023–12, 2023. find
  6. C. Wang, J. Ong, C. Wang, H. Ong, R. Cheng, and D. Ong, “Potential for gpt technology to optimize future clinical decision-making using retrieval-augmented generation,” Annals of Biomedical Engineering, pp. 1–4, 2023. find
  7. L. Loukas, I. Stogiannidis, O. Diamantopoulos, P. Malakasiotis, and S. Vassos, “Making llms worth every penny: Resource-limited text classification in banking,” in ICAIF, 2023. find
  8. A. Kuppa, N. Rasumov-Rahe, and M. Voses, “Chain of reference prompting helps llm to think like a lawyer.” find
  9. R. Z. Mahari, “Autolaw: Augmented legal reasoning through legal precedent prediction,” arXiv, 2021. find
  10. V. Kumar, L. Gleyzer, A. Kahana, K. Shukla, and G. E. Karniadakis, “Mycrunchgpt: A llm assisted framework for scientific machine learning,” Journal of Machine Learning for Modeling and Computing, vol. 4, no. 4, 2023. find
  11. J. Boyko, J. Cohen, N. Fox, M. H. Veiga, J. I. Li, J. Liu, B. Modenesi, A. H. Rauch, K. N. Reid, S. Tribedi et al., “An interdisciplinary outlook on large language models for scientific research,” arXiv, 2023. find
  12. M. H. Prince, H. Chan, A. Vriza, T. Zhou, V. K. Sastry, M. T. Dearing, R. J. Harder, R. K. Vasudevan, and M. J. Cherukara, “Opportunities for retrieval and tool augmented large language models in scientific facilities,” arXiv, 2023. find
  13. V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering,” in EMNLP, 2020.
  14. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” NeurIPS, 2020.
  15. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark et al., “Improving language models by retrieving from trillions of tokens,” in ICML, 2022. find
  16. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du et al., “Lamda: Language models for dialog applications,” arXiv, 2022.
  17. N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych, “Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models,” in NeurIPS, 2021. find
  18. I. Soboroff, S. Huang, and D. Harman, “Trec 2019 news track overview.” in TREC, 2019. find
  19. E. Voorhees, T. Alam, S. Bedrick, D. Demner-Fushman, W. R. Hersh, K. Lo, K. Roberts, I. Soboroff, and L. L. Wang, “Trec-covid: constructing a pandemic information retrieval test collection,” in ACM SIGIR Forum, vol. 54, no. 1, 2021, pp. 1–12. find
  20. “Chatgpt knowledge retrieval,” https://platform.openai.com/docs/assistants/tools/knowledge-retrieval, 2023. ↗
  21. J. Liu, “LlamaIndex,” 11 2022. [Online]. Available: https://github.com/jerryjliu/llama_index ↗
  22. “Chatrtx,” https://www.nvidia.com/en-us/ai-on-rtx/chatrtx/. ↗
  23. “LangChain,” https://www.langchain.com/. ↗
  24. S. Semnani, V. Yao, H. Zhang, and M. Lam, “WikiChat: Stopping the hallucination of large language model chatbots by few-shot grounding on Wikipedia,” in EMNLP, 2023. find
  25. “Bing search,” https://www.microsoft.com/en-us/bing?form=MG0AUO&OCID=MG0AUO\#faq, 2024. ↗
  26. A. Lozano, S. L. Fleming, C.-C. Chiang, and N. Shah, “Clinfo. ai: An open-source retrieval-augmented large language model system for answering medical questions using scientific literature,” in PACIFIC SYMPO-SIUM ON BIOCOMPUTING 2024, 2023. find
  27. “Generative ai in search: Let google do the searching for you,” https://blog.google/products/search/generative-ai-google-search-may-2024/. ↗
  28. “Perplexity ai,” https://www.perplexity.ai//. ↗
  29. N. Shinn, B. Labash, and A. Gopinath, “Reflexion: an autonomous agent with dynamic memory and self-reflection,” arXiv preprint arXiv:2303.11366, 2023.
  30. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” arXiv, 2022.
  31. A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “Self-rag: Learning to retrieve, generate, and critique through self-reflection,” in ICLR, 2024.
  32. G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave, “Unsupervised dense information retrieval with contrastive learning,” Transactions on Machine Learning Research, 2022.
  33. L. Xiong, C. Xiong, Y. Li, K.-F. Tang, J. Liu, P. N. Bennett, J. Ahmed, and A. Overwijk, “Approximate nearest neighbor negative contrastive learning for dense text retrieval,” in ICLR, 2021.
  34. Z. Peng, X. Wu, and Y. Fang, “Soft prompt tuning for augmenting dense retrieval with large language models,” arXiv, 2023. find
  35. N. Kassner and H. Schütze, “Bert-knn: Adding a knn search component to pretrained language models for better qa,” in Findings of ACL: EMNLP, 2020. find
  36. V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering,” in EMNLP, 2020.
  37. N. Carlini, M. Jagielski, C. A. Choquette-Choo, D. Paleka, W. Pearce, H. Anderson, A. Terzis, K. Thomas, and F. Tramèr, “Poisoning web-scale training datasets is practical,” arXiv, 2023. find
  38. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee et al., “Natural questions: a benchmark for question answering research,” TACL, vol. 7, pp. 452– 466, 2019.
  39. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning, “Hotpotqa: A dataset for diverse, explainable multi-hop question answering,” in EMNLP, 2018.
  40. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng, “Ms marco: A human generated machine reading comprehension dataset,” choice, vol. 2640, p. 660, 2016.
  41. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv, 2023.
  42. Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” arXiv, 2024.
  43. Z. Zhong, Z. Huang, A. Wettig, and D. Chen, “Poisoning retrieval corpora by injecting adversarial passages,” in EMNLP, 2023. find
  44. N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, “Baseline defenses for adversarial attacks against aligned language models,” arXiv, 2023.
  45. G. Alon and M. Kamfonas, “Detecting language model attacks with perplexity,” arXiv, 2023.
  46. H. Gonen, S. Iyer, T. Blevins, N. A. Smith, and L. Zettlemoyer, “Demystifying prompts in language models via perplexity estimation,” arXiv, 2022.
  47. F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” in NeurIPS ML Safety Workshop, 2022.
  48. Y. Liu, G. Deng, Y. Li, K. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng, and Y. Liu, “Prompt injection attack against llm-integrated applications,” arXiv, 2023.
  49. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” in AISec, 2023. — Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
  50. R. Pedro, D. Castro, P. Carreira, and N. Santos, “From prompt injections to sql injection attacks: How protected is your llm-integrated web application?” arXiv, 2023. find
  51. H. J. Branch, J. R. Cefalu, J. McHugh, L. Hujer, A. Bahl, D. d. C. Iglesias, R. Heichman, and R. Darwishi, “Evaluating the susceptibility of pre-trained language models via handcrafted adversarial examples,” arXiv, 2022. find
  52. A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” in NeurIPS, 2023.
  53. A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv, 2023.
  54. G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang, T. Zhang, and Y. Liu, “Masterkey: Automated jailbreaking of large language model chatbots,” in NDSS, 2024. find
  55. X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal, “Visual adversarial examples jailbreak aligned large language models,” arXiv, 2023.
  56. H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y. Song, “Multi-step jailbreaking privacy attacks on chatgpt,” arXiv, 2023.
  57. X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, “"do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models,” arXiv, 2023.
  58. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., “Extracting training data from large language models,” in Usenix Security, 2021.
  59. N. Kandpal, M. Jagielski, F. Tramèr, and N. Carlini, “Backdoor attacks for in-context learning with language models,” in ICML Workshop, 2023. find
  60. A. Wan, E. Wallace, S. Shen, and D. Klein, “Poisoning language models during instruction tuning,” in ICML, 2023. find
  61. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang, “Quantifying memorization across neural language models,” in ICLR, 2022. find
  62. N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel, “Extracting training data from large language models,” in Usenix Security, 2021.
  63. J. Mattern, F. Mireshghallah, Z. Jin, B. Schoelkopf, M. Sachan, and T. Berg-Kirkpatrick, “Membership inference attacks against language models via neighbourhood comparison,” in Findings of ACL: EMNLP, 2023.
  64. X. Pan, M. Zhang, S. Ji, and M. Yang, “Privacy risks of general-purpose language models,” in IEEE S & P, 2020. find
  65. “Exploring prompt injection attacks,” https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/. ↗
  66. B. Biggio, B. Nelson, and P. Laskov, “Poisoning attacks against support vector machines,” in ICML, 2012. find
  67. T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identifying vulnerabilities in the machine learning model supply chain,” IEEE Access, 2017.
  68. Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang, “Trojaning attack on neural networks,” in NDSS, 2018. find
  69. X. Chen, C. Liu, B. Li, K. Lu, and D. Song, “Targeted backdoor attacks on deep learning systems using data poisoning,” arXiv, 2017. find
  70. A. Shafahi, W. R. Huang, M. Najibi, O. Suciu, C. Studer, T. Dumitras, and T. Goldstein, “Poison frogs! targeted clean-label poisoning attacks on neural networks,” in NeurIPS, 2018. find
  71. E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” in AISTATS, 2020. find
  72. M. Fang, X. Cao, J. Jia, and N. Gong, “Local model poisoning attacks to byzantine-robust federated learning,” in USENIX Security Symposium, 2020. find
  73. Z. Zhang, J. Jia, B. Wang, and N. Z. Gong, “Backdoor attacks to graph neural networks,” in SACMAT, 2021. find
  74. J. Jia, Y. Liu, and N. Z. Gong, “Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning,” in IEEE S & P, 2022. find
  75. X. Li and J. Li, “Angle-optimized text embeddings,” arXiv preprint arXiv:2309.12871, 2023.
  76. “Whereisai/uae-large-v1,” https://huggingface.co/WhereIsAI/UAE-Large-V1. ↗
  77. A. Kerckhoffs, “La cryptographie militaire,” Journal des sciences militaires, 1883. find
  78. J. Ebrahimi, A. Rao, D. Lowd, and D. Dou, “Hotflip: White-box adversarial examples for text classification,” in ACL, 2018.
  79. J. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, and Y. Qi, “Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp,” in EMNLP, 2020.
  80. D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits, “Is bert really robust? a strong baseline for natural language attack on text classification and entailment,” in AAAI, 2020.
  81. J. Li, S. Ji, T. Du, B. Li, and T. Wang, “Textbugger: Generating adversarial text against real-world applications,” in NDSS, 2019.
  82. L. Li, R. Ma, Q. Guo, X. Xue, and X. Qiu, “Bert-attack: Adversarial attack against bert using bert,” in EMNLP, 2020.
  83. J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi, “Black-box generation of adversarial text sequences to evade deep learning classifiers,” in SPW, 2018.
  84. “Ms-marco microsoft bing,” https://microsoft.github.io/msmarco/. ↗
  85. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” 2023. find
  86. N. Carlini and A. Terzis, “Poisoning and backdooring contrastive learning,” in ICLR, 2022. find
  87. N. Carlini, “Poisoning the unlabeled dataset of semi-supervised learning,” in USENIX Security, 2021. find
  88. H. Liu, J. Jia, and N. Z. Gong, “Poisonedencoder: Poisoning the unlabeled pre-training data in contrastive learning,” in USENIX Security Symposium, 2022. find
  89. M. R. Rizqullah, A. Purwarianti, and A. F. Aji, “Qasina: Religious domain question answering using sirah nabawiyah,” in ICAICTA, 2023. find
  90. Y. Huang, S. Gupta, M. Xia, K. Li, and D. Chen, “Catastrophic jailbreak of open-source llms via exploiting generation,” arXiv, 2023.
  91. Y. Du, A. Bosselut, and C. D. Manning, “Synthetic disinformation attacks on automated fact verification systems,” in AAAI, 2022. find
  92. Y. Pan, L. Pan, W. Chen, P. Nakov, M.-Y. Kan, and W. Y. Wang, “On the risk of misinformation pollution with large language models,” in EMNLP, 2023.
  93. S.-Q. Yan, J.-C. Gu, Y. Zhu, and Z.-H. Ling, “Corrective retrieval augmented generation,” arXiv, 2024.
  94. O. Yoran, T. Wolfson, O. Ram, and J. Berant, “Making retrieval-augmented language models robust to irrelevant context,” in ICLR, 2023.
  95. H. Luo, T. Zhang, Y.-S. Chuang, Y. Gong, Y. Kim, X. Wu, H. Meng, and J. Glass, “Search augmented instruction learning,” in EMNLP, 2023, pp. 3717–3729. find
  96. T. Zhang, S. G. Patil, N. Jain, S. Shen, M. Zaharia, I. Stoica, and J. E. Gonzalez, “Raft: Adapting language model to domain specific rag,” arXiv, 2024.
  97. J. Steinhardt, P. W. W. Koh, and P. S. Liang, “Certified defenses for data poisoning attacks,” NeurIPS, 2017. find
  98. B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao, “Neural cleanse: Identifying and mitigating backdoor attacks in neural networks,” in IEEE S& P, 2019. find
  99. J. Jia, X. Cao, and N. Z. Gong, “Intrinsic certified robustness of bagging against data poisoning attacks,” in AAAI, 2021. find
  100. J. Jia, Y. Liu, X. Cao, and N. Z. Gong, “Certified robustness of nearest neighbors against data poisoning and backdoor attacks,” in AAAI, 2022. find
  101. J. Jia, Y. Liu, Y. Hu, and N. Z. Gong, “Pore: Provably robust recommender systems against data poisoning attacks,” in USENIX Security Symposium, 2023. find
  102. Y. Wang, W. Zou, and J. Jia, “Fcert: Certifiably robust few-shot classification in the era of foundation models,” in IEEE S & P, 2024. find
  103. S. Shan, A. N. Bhagoji, H. Zheng, and B. Y. Zhao, “Poison forensics: Traceback of data poisoning attacks in neural networks,” in USENIX Security Symposium, 2022. find
  104. F. Jelinek, “Interpolated estimation of markov source parameters from sparse data,” in Proc. Workshop on Pattern Recognition in Practice, 1980. find
  105. “Tiktoken,” https://github.com/openai/tiktoken. ↗
  106. J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “Fever: a large-scale dataset for fact extraction and verification,” arXiv, 2018.
  107. E. Kortukov, A. Rubinstein, E. Nguyen, and S. J. Oh, “Studying large language model behaviors under realistic knowledge conflicts,” arXiv, 2024. Algorithm 2: PoisonedRAG (white-box) Input: M target questions Q 1 , Q 2 , · · · , Q M , target answers R 1 , R 2 , · · · , R M , hyperparameters N, L, V ,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zou-2024-poisonedrag-knowledge-corruption-attacks-retrieval/