Elamur Research⌕
☰
home / papers / 2026 / Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)

Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)

Paper · 2026 · Junki Mori, Kazuya Kakizaki, Taiki Miyagawa et al.

Summary

Retrieval-Augmented Generation (RAG) systems built on sensitive databases can leak private details, such as patient names, to attackers and even to ordinary users. Earlier differentially private RAG methods add noise to every answer, so the privacy cost grows with each query. Mori et al. propose DP-SynRAG, which uses LLMs to build a synthetic copy of the RAG database under differential privacy (DP) before any queries arrive. Because later retrieval over this copy counts as post-processing, the privacy budget stays fixed no matter how many queries follow.

The method first groups documents with DP-protected soft clustering based on keywords and document-level embeddings. It then runs private prediction, which samples tokens from summed, clipped logits as an exponential mechanism, separately within each cluster. This keeps topic-specific facts that plain private prediction tends to average away. An LLM self-filtering step then removes weak synthetic texts.

On Medical Synth, Movielens and SearchQA, DP-SynRAG outperforms query-level private RAG in most settings and nearly eliminates name leakage. This makes reusable private knowledge bases practical.

Key points

  • The experiments use three datasets (Medical Synth, Movielens, SearchQA) and three LLMs (Phi-4-mini, Gemma-2-2B, Llama-3.1-8B). The main baseline is DP-RAG, which applies per-query DP.
  • Leakage was tested on Medical Synth with ε_total = 10, using 1,000 benign queries and 100 attack queries asking which patient had a given disease.
    • Non-private RAG leaked full names 4–22 times under benign queries and 81–90 times under attack.
    • DP-SynRAG kept leakage at 0–2.4 for benign queries and 0.25–1.25 under attack.
  • In the ablation, replacing soft clustering with hard clustering (L = 1) cut Medical Synth accuracy by about 16–32 points. Removing self-filtering or embedding-based retrieval usually caused smaller drops.
  • Limitations:
    • The method fails on rare topics backed by about 30 or fewer documents.
    • Utility suffers under very tight privacy budgets.
    • The synthetic database must be regenerated when the source data changes, though no model retraining is needed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬RIKEN Center for Advanced Intelligence ProjectTokyo · Japan15 works · 957 citations🏢NECTokyo · Japan6 works · 546 citations🏢NEC (United States)Irving · United States6 works · 539 citations🎓Institute of Science TokyoTokyo · Japan10 works · 20 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Information retrieval & search · 473 worksText generation · 202 works

Cite

Cite as Junki Mori, Kazuya Kakizaki, Taiki Miyagawa and Jun Sakuma. 2026. Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG). Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.62

@inproceedings{mori2026differentially,
  title = {{Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)}},
  author = {Junki Mori, Kazuya Kakizaki, Taiki Miyagawa and Jun Sakuma},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.62},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.62}
}

References

43 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kareem Amin, Salman Avestimehr, Sara Babakniya, Alex Bie, Weiwei Kong, Natalia Ponomareva, and Umar Syed. 2025. Clustering and median aggregation improve differentially private inference. Preprint, arXiv:2506.04566. — [arXiv:2506.04566](https://arxiv.org/abs/2506.04566) find
  2. Kareem Amin, Alex Bie, Weiwei Kong, Alexey Kurakin, Natalia Ponomareva, Umar Syed, Andreas Terzis, and Sergei Vassilvitskii. 2024. Private prediction for large-scale synthetic text generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7244–7262, Miami, Florida,… find
  3. Maya Anderson, Guy Amit, and Abigail Goldsteen. 2025. Is my data in your retrieval database? membership inference attacks against retrieval augmented generation. In Proceedings of the 11th International Conference on Information Systems Security and Privacy, page 474–485. SCITEPRESS - Science and Te… find
  4. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.
  5. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.". find
  6. Mark Bun and Thomas Steinke. 2016. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography, pages 635– 658, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  7. Mark Cesar and Ryan Rogers. 2021. Bounding, concentrating, and truncating: Unifying privacy loss composition for data analytics. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, volume 132 of Proceedings of Machine Learning Research, pages 421–457. PMLR. find
  8. Sai Chen, Fengran Mo, Yanhao Wang, Cen Chen, Jian-Yun Nie, Chengyu Wang, and Jamie Cui. 2023. A customized text sanitization mechanism with differential privacy. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5747– 5758, Toronto, Canada. Association for Computational L… find
  9. Yujin Choi, Youngjoo Park, Junyoung Byun, Jaewook Lee, and Jinseong Park. 2025. Safeguarding privacy of retrieval data against membership inference attacks: Is this query too close to home? Preprint, arXiv:2505.22061. find
  10. Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. Preprint, arXiv:1704.05179.
  11. James Flemings, Meisam Razaviyayn, and Murali Annavaram. 2024. Differentially private next-token prediction of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pape… find
  12. Fengyu Gao, Ruida Zhou, Tianhao Wang, Cong Shen, and Jing Yang. 2025. Data-adaptive differentially private prompt synthesis for in-context learning. In International Conference on Representation Learning, volume 2025, pages 60152–60180. find
  13. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. Preprint, arXiv:2312.10997.
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  15. Nicolas Grislain. 2025. Rag with differential privacy. In 2025 IEEE Conference on Artificial Intelligence (CAI), pages 847–852. find
  16. F. Maxwell Harper and Joseph A. Konstan. 2015. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5(4). find
  17. Jiaming He, Cheng Liu, Guanyu Hou, Wenbo Jiang, and Jiachen Li. 2025. Press: Defending privacy in retrieval-augmented generation via embedding space shifting. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. find
  18. Ali Naseh, Yuefeng Peng, Anshuman Suri, Harsh Chaudhari, Alina Oprea, and Amir Houmansadr. 2025. Riddle me this! stealthy membership inference for retrieval-augmented generation. Preprint, arXiv:2502.00306. — [arXiv:2502.00306](https://arxiv.org/abs/2502.00306) find
  19. Junyuan Hong, Jiachen T. Wang, Chenhui Zhang, Zhangheng LI, Bo Li, and Zhangyang Wang. 2024. DP-OPT: Make large language model your privacy-preserving prompt engineer. In The Twelfth International Conference on Learning Representations. find
  20. Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. 2007. Smooth sensitivity and sampling in private data analysis. In Proceedings of the Thirty-Ninth Annual ACM Symposium on Theory of Computing, STOC ’07, page 75–84, New York, NY, USA. Association for Computing Machinery. find
  21. Changyue Jiang, Xudong Pan, Geng Hong, Chenfu Bao, Yang Chen, and Min Yang. 2025. Feedback-guided extraction of knowledge base from retrieval-augmented llm applications. Preprint, arXiv:2411.14110. — [arXiv:2411.14110](https://arxiv.org/abs/2411.14110) find
  22. Yuefeng Peng, Junda Wang, Hong Yu, and Amir Houmansadr. 2025. Data extraction attacks in retrieval-augmented generation via backdoors. Preprint, arXiv:2411.01705. — [arXiv:2411.01705](https://arxiv.org/abs/2411.01705) find
  23. Tatsuki Koga, Ruihan Wu, and Kamalika Chaudhuri. 2025. Privacy-preserving retrieval-augmented generation with differential privacy. Preprint, arXiv:2412.04697. — [arXiv:2412.04697](https://arxiv.org/abs/2412.04697) find
  24. Zhenting Qi, Hanlin Zhang, Eric P. Xing, Sham M. Kakade, and Himabindu Lakkaraju. 2025. Follow my instruction and spill the beans: Scalable data extraction from retrieval-augmented generation systems. In The Thirteenth International Conference on Learning Representations. find
  25. Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, and Andreas Terzis. 2024. Harnessing large-language models to generate private synthetic text. Preprint, arXiv:2306.01684. — [arXiv:2306.01684](https://arxiv.org/abs/2306.01684) find
  26. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Proc…
  27. Yuying Li, Gaoyang Liu, Chen Wang, and Yang Yang. 2025. Generating is believing: Membership inference attacks against retrieval-augmented generation. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. find
  28. Mingrui Liu, Sixiao Zhang, and Cheng Long. 2025. Mask-based membership inference attacks for retrieval-augmented generation. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 2894–2907, New York, NY, USA. Association for Computing Machinery. find
  29. Christian Di Maio, Cristian Cosci, Marco Maggini, Valentina Poggioni, and Stefano Melacci. 2024. Pirates of the rag: Adaptively attacking llms to leak knowledge bases. Preprint, arXiv:2412.18295. — [arXiv:2412.18295](https://arxiv.org/abs/2412.18295) find
  30. Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, and 57 others. 2025. Phi-4-mini technical…
  31. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  32. Josh Smith, Hassan Jameel Asghar, Gianpaolo Gioiosa, Sirine Mrabet, Serge Gaspers, and Paul Tyler. 2022. Making the most of parallel composition in differential privacy. In Proceedings on Privacy Enhancing Technologies Symposium, page 253–273. find
  33. Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. 2024. Privacy-preserving in-context learning with differentially private few-shot generation. In The Twelfth International Conference on Learning Represe… find
  34. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179… find
  35. Haoran Wang, Xiongxiao Xu, Baixiang Huang, and Kai Shu. 2025a. Privacy-aware decoding: Mitigating privacy leakage of large language models in retrieval-augmented generation. Preprint, arXiv:2508.03098. — [arXiv:2508.03098](https://arxiv.org/abs/2508.03098) find
  36. Yuhao Wang, Wenjie Qu, Yanze Jiang, Zichen Liu, Yue Liu, Shengfang Zhai, Yinpeng Dong, and Jiaheng Zhang. 2025b. Silent leaks: Implicit knowledge extraction attack on rag systems through benign queries. Preprint, arXiv:2505.15420. — [arXiv:2505.15420](https://arxiv.org/abs/2505.15420) find
  37. Chulin Xie, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, Haotian Jiang, Huishuai Zhang, Yin Tat Lee, Bo Li, and Sergey Yekhanin. 2024. Differentially private synthetic data via foundation model APIs 2: Text. In Proceedings of the 41st International Conference on Mac… find
  38. Dixi Yao and Tian Li. 2025. Differentially private retrieval augmented generation with random projection. In ICLR 2025 Workshop on Building Trust in Language Models and Applications. find
  39. Da Yu, Peter Kairouz, Sewoong Oh, and Zheng Xu. 2024. Privacy-preserving instructions for aligning large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 57480–57506. PMLR. find
  40. Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, and Sherman S. M. Chow. 2021. Differential privacy for text analytics via natural text sanitization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3853–3866, Online. Association for Computational Linguis… find
  41. Xiang Yue, Huseyin Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim. 2023. Synthetic text generation with differential privacy: A simple and practical recipe. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguist… find
  42. Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang. 2024. The good and the bad: Exploring privacy issues in retrieval-augmented generation (RAG). In Findings of the Association for Computational Linguistics: ACL 202… find
  43. Shenglai Zeng, Jiankun Zhang, Pengfei He, Jie Ren, Tianqi Zheng, Hanqing Lu, Han Xu, Hui Liu, Yue Xing, and Jiliang Tang. 2025. Mitigating the privacy issues in retrieval-augmented generation (rag) via pure synthetic data. Preprint, arXiv:2406.14773. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mori-2026-differentially-private-synthetic-text-generation/