Elamur Research⌕
☰
home / papers / 2026 / LeakDojo: Decoding the Leakage Threats of RAG Systems

LeakDojo: Decoding the Leakage Threats of RAG Systems

Paper · 2026 · Maosen Zhang, Jianshuo Dong, Lu Boting et al.

Summary

RAG systems pair LLMs with curated knowledge bases that are expensive to build. This makes them targets for leakage attacks, where an attacker with only black-box access tries to pull out as many stored chunks as possible. Earlier attack papers used different RAG setups and query budgets, so their results cannot be compared directly. Those setups also rarely included modern enhancement modules or defenses.

The authors built LeakDojo, a modular, open-source framework. It treats the RAG pipeline, the attack (anchor query generator plus adversarial instruction) and the defenses (input- and output-stage) as separately configurable parts. With it they re-evaluated six published attacks across 14 LLMs, four English datasets, and RAG variants with a reranker, rewriter or summarizer.

They report three main findings:

  • Query generation and adversarial instructions affect leakage independently, and total leakage roughly equals the product of the two.
  • LLMs that follow instructions better tend to leak more.
  • Modules that make answers more faithful to retrieved content can also increase leakage.

These results give RAG builders concrete guidance when choosing models and components. The study is English-only, so leakage behaviour in other languages, such as Ukrainian, has not been tested.

Key points

  • Attacks benchmarked under one protocol: TGTB, PIDE, DGEA, RAG-Thief, PoR and IKEA, each limited to N = 200 interaction rounds.
  • Models tested: 14 LLMs, six primary and eight supplementary, both open and closed. Examples include the Qwen-2.5 family, Qwen-3-235B, DeepSeek-V3, Kimi-K2, GPT-5.1, o4-mini, Gemini-2.5-flash, Gemini-3-flash and Gemma-3-27B.
  • Instruction-following and leakage: IFEval strict score correlates positively with leakage probability (Pearson r = 0.578, p = 0.039).
  • Configurable components: retrieval settings (embedding model, similarity threshold, top-k), optional rewriter, reranker and summarizer modules, and defense modules at both input and output. A case study adds new attack implementations, including one the authors call CodeClaim.
  • Limitations: only English corpora were used (including healthcare and personal-email data), and attack cost such as tokens and latency was not measured. The code is released on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Maosen Zhang, Jianshuo Dong, Lu Boting, Li Wenyue, Xiaoping Zhang, Tianwei Zhang and Han Qiu. 2026. LeakDojo: Decoding the Leakage Threats of RAG Systems. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.287

@inproceedings{zhang2026leakdojo,
  title = {{LeakDojo: Decoding the Leakage Threats of RAG Systems}},
  author = {Maosen Zhang, Jianshuo Dong, Lu Boting, Li Wenyue, Xiaoping Zhang, Tianwei Zhang and Han Qiu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.287},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.287}
}

References

46 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Divyansh Agarwal, Alexander Richard Fabbri, Ben Risher, Philippe Laban, Shafiq Joty, and Chien-Sheng Wu. 2024. Prompt leakage effect and mitigation strategies for multi-turn llm applications. In EMNLP: Industry Track. find
  2. Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. find
  3. Jaime Carbonell and Jade Goldstein. 1998. The use of https://aclrollingreview.org/ responsibleNLPresearch/ mmr, diversity-based reranking for reordering documents and producing summaries. In SIGIR. ↗
  4. Stav Cohen, Ron Bitton, and Ben Nassi. 2024. Unleashing worms and extracting data: Escalating the outcome of attacks against rag-based inference in scale and severity using jailbreaking. arXiv preprint arXiv:2409.08045. — [arXiv:2409.08045](https://arxiv.org/abs/2409.08045) find
  5. Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tramèr. 2025. Defeating prompt injections by design. arXiv preprint arXiv:2503.18813. — [arXiv:2503.18813](https://arxiv.org/abs/2503.18813) find
  6. Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In NeurIPS Datasets and Benchmarks Track. find
  7. Christian Di Maio, Cristian Cosci, Marco Maggini, Valentina Poggioni, and Stefano Melacci. 2024. Pirates of the rag: Adaptively attacking llms to leak knowledge bases. arXiv preprint arXiv:2412.18295. — [arXiv:2412.18295](https://arxiv.org/abs/2412.18295) find
  8. Jianshuo Dong, Yutong Zhang, Liu Yan, Zhenyu Zhong, Tao Wei, Ke Xu, Minlie Huang, Chao Zhang, and Han Qiu. 2025. “i’ve decided to leak”: Probing internals behind prompt leakage intents. In EMNLP. find
  9. Enron. 2015. Enron email dataset. https://www.cs.cmu.edu/~enron/. ↗
  10. ExplodingGradients. 2024. Ragas: Supercharge your llm application evaluations. https://github.com/ explodinggradients/ragas. ↗
  11. Yunfan Gao, Yun Xiong, Meng Wang, and Haofen Wang. 2024. Modular rag: Transforming rag systems into lego-like reconfigurable frameworks. arXiv preprint arXiv:2407.21059. — [arXiv:2407.21059](https://arxiv.org/abs/2407.21059) find
  12. Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. 2024. Does fine-tuning llms on new knowledge encourage hallucinations? EMNLP. find
  13. Jun Guo, Bojian Chen, Zhichao Zhao, Jindong He, Shichun Chen, Donglan Hu, and Hao Pan. 2024. Bkrag: A bge reranker rag for similarity analysis of power project requirements. In PRIS. find
  14. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In ICML.
  15. Lei Huang, Xiaocheng Feng, Weitao Ma, Yuchun Fan, Xiachong Feng, Yuxuan Gu, Yangfan Ye, Liang Zhao, Weihong Zhong, Baoxin Wang, and 1 others. 2025. Alleviating hallucinations from knowledge misalignment in large language models via selective abstention learning. In ACL. find
  16. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys.
  17. Changyue Jiang, Xudong Pan, Geng Hong, Chenfu Bao, and Min Yang. 2024. Feedback-guided extraction of knowledge base from retrieval-augmented llm applications. arXiv preprint arXiv:2411.14110. — [arXiv:2411.14110](https://arxiv.org/abs/2411.14110) find
  18. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In SIGIR.
  19. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPS.
  20. Zhonghao Li, Xuming Hu, Aiwei Liu, Kening Zheng, Sirui Huang, and Hui Xiong. 2024. Refiner: Restructure retrieval content efficiently to advance question-answering capabilities. arXiv preprint arXiv:2406.11357. — [arXiv:2406.11357](https://arxiv.org/abs/2406.11357) find
  21. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In USENIX Security.
  22. Peizhuo Lv, Mengjie Sun, Hao Wang, Xiaofeng Wang, Shengzhi Zhang, Yuxuan Chen, Kai Chen, and Limin Sun. 2025. Rag-wm: An efficient black-box watermarking approach for retrieval-augmented generation of large language models. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communicati… find
  23. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In EMNLP.
  24. Shengyu Mao, Yong Jiang, Boli Chen, Xiao Li, Peng Wang, Xinyu Wang, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. 2024. Rafe: Ranking feedback improves query rewriting for rag. arXiv preprint arXiv:2405.14431. find
  25. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, and 1 others. 2024. Harmbench: a standardized evaluation framework for automated red teaming and robust refusal. In ICML.
  26. Financial Opinion Mining and Qeustion Answering. 2018. Fiqa-2018. find
  27. Karl Pearson. 1901. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science. find
  28. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. In NeurIPS ML Safety Workshop.
  29. Zhenting Qi, Hanlin Zhang, Eric Xing, Sham Kakade, and Himabindu Lakkaraju. 2024. Follow my instruction and spill the beans: Scalable data extraction from retrieval-augmented generation systems. arXiv preprint arXiv:2402.17840. — [arXiv:2402.17840](https://arxiv.org/abs/2402.17840) find
  30. David Rau, Shuai Wang, Hervé Déjean, and Stéphane Clinchant. 2024. Context embeddings for efficient answer generation in rag. arXiv preprint arXiv:2407.09252. — [arXiv:2407.09252](https://arxiv.org/abs/2407.09252) find
  31. Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021. “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai. In CHI. find
  32. Spurthi Setty, Harsh Thakkar, Alyssa Lee, Eden Chung, and Natan Vidra. 2024. Improving retrieval for rag based question answering models on financial documents. arXiv preprint arXiv:2404.07221. — [arXiv:2404.07221](https://arxiv.org/abs/2404.07221) find
  33. Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Yinxiao Liu, Simon Tong, Jindong Chen, and Lei Meng. 2024. Rewritelm: An instruction-tuned large language model for text rewriting. In AAAI. find
  34. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663.
  35. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. find
  36. Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, and 1 others. 2024. The good and the bad: Exploring privacy issues in retrieval-augmented generation (rag). In ACL (Findings). find
  37. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. In ACL (Findings).
  38. Yiming Zhang, Nicholas Carlini, and Daphne Ippolito. 2024. Effective prompt extraction from language models. In COLM. find
  39. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
  40. Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, and 1 others. 2024. Realm: Rag-driven enhancement of multimodal electronic health records analysis via large language models. arXiv preprint arXiv:2402.07016. — [arXiv:2402.07016](https://arxiv.org/abs/2402.07016) find
  41. Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, and Stuart Russell. 2024. Tensor trust: Interpretable prompt injection attacks from an online game. In ICLR. find
  42. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. arXiv preprint arXiv:2004.14974.
  43. Junlin Wang, Tianyi Yang, Roy Xie, and Bhuwan Dhingra. 2024. Raccoon: Prompt extraction benchmark of llm-integrated applications. In ACL. find
  44. Yuhao Wang, Wenjie Qu, Shengfang Zhai, Yanze Jiang, Zichen Liu, Yue Liu, Yinpeng Dong, and Jiaheng Zhang. 2025. Silent leaks: Implicit knowledge extraction attack on rag systems through benign queries. arXiv preprint arXiv:2505.15420. — [arXiv:2505.15420](https://arxiv.org/abs/2505.15420) find
  45. Nirmalie Wiratunga, Ramitha Abeyratne, Lasal Jayawardena, Kyle Martin, Stewart Massie, Ikechukwu Nkisi-Orji, Ruvan Weerasinghe, Anne Liret, and Bruno Fleisch. 2024. Cbr-rag: case-based reasoning for retrieval augmented generation in llms for legal question answering. In International Conference on C…
  46. Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. 2024. Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410.13085. — [arXiv:2410.13085](https://arxiv.org/abs/2410.13085) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-leakdojo-decoding-leakage-threats-rag/