Elamur Research⌕
☰
home / papers / 2026 / DP³: Differentially Private Prompt Perturbation for Multi-turn LLM Inference

DP³: Differentially Private Prompt Perturbation for Multi-turn LLM Inference

Paper · 2026 · Lele Zheng, Chao Zhang, Feiyang Yuan et al.

Summary

Prompts sent to black-box LLM APIs can expose sensitive data. A common defence is client-side perturbation under local differential privacy (ε-LDP). Existing token-substitution schemes such as SANTEXT, CUSTEXT, InferDPT and CAPE handle each query in isolation, and this fails in conversations. Tokens that reappear across turns get re-randomized every time, so sequential composition makes the privacy cost grow. Independent substitutions also weaken the semantic links between turns.

DP$^3$ is an ε-LDP framework built for multi-turn inference. It has three parts:

  • A perturbation mapping table (PMT) lets a repeated token reuse its earlier replacement.
  • A utility function ranks candidate substitutes by combining embedding distance with attention-derived context from a small local model.
  • A two-stage bucketed exponential mechanism samples replacements while limiting the long tail of a large candidate set.

On multi-turn dialogue benchmarks, including MT-Bench-101, the authors report a better privacy–utility balance and stronger resistance to inference attacks than the baselines. This matters for using chat assistants on confidential inputs when model internals are inaccessible.

Key points

  • PMT ablation (MT-Bench-101, ε = 1, K = 5):
    • DP$^3$ accuracy rises from 1.99% to 18.33% with PMT.
    • Adding PMT to baselines helps little: SANTEXT goes from 15.83% to 18.10%, CUSTEXT from 3.62% to 4.07%, and InferDPT does not change.
  • Bucket size: K was varied over {5, 10, 20, 100, 200}. Smaller buckets gave clearly higher utility.
  • Client-side cost:
    • DP$^3$ runs a local BERT-base model (~110M parameters).
    • It uses about 31.44 GB of RAM and 5767.1 MiB of VRAM.
    • Perturbation takes 58.21% of end-to-end time, compared with 12.38% for CAPE and 82.71–95.54% for SANTEXT, CUSTEXT and InferDPT.
  • Limitations:
    • PMT depends on identical token forms, so paraphrasing or language switching reduces its benefit.
    • A poor first substitution propagates to later turns.
    • Deterministic mappings may leak frequency patterns over long histories.
  • Code: github.com/XidianNSS/DP3.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Xidian UniversityXi'an · China17 works · 1,357 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Lele Zheng, Chao Zhang, Feiyang Yuan, Ke Cheng, Tao Zhang, Anxiao Song and Yulong Shen. 2026. DP³: Differentially Private Prompt Perturbation for Multi-turn LLM Inference. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.924

@inproceedings{zheng2026differentially,
  title = {{DP³: Differentially Private Prompt Perturbation for Multi-turn LLM Inference}},
  author = {Lele Zheng, Chao Zhang, Feiyang Yuan, Ke Cheng, Tao Zhang, Anxiao Song and Yulong Shen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.924},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.924}
}

References

32 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad Ahmadian and Dan C Marinescu. 2018. Information leakage in cloud data warehouses. IEEE Transactions on Sustainable Computing, 5(2):192– 203. find
  2. Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024. MT-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Associatio…
  3. Mihir Bellare and Phillip Rogaway. 2005. Introduction to modern cryptography. Ucsd Cse, 207:207. find
  4. Zvika Brakerski, Craig Gentry, and Vinod Vaikuntanathan. 2014. (leveled) fully homomorphic encryption without bootstrapping. ACM Transactions on Computation Theory (TOCT), 6(3):1–36. find
  5. Zvika Brakerski and Vinod Vaikuntanathan. 2014. Efficient fully homomorphic encryption from (standard) lwe. SIAM Journal on computing, 43(2):831–871. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  7. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21)…
  8. Jiawei Chen, Xinyan Guan, Qianhao Yuan, Mo Guozhao, Weixiang Zhou, Yaojie Lu, Hongyu Lin, Ben He, Le Sun, and Xianpei Han. 2025. Consistentchat: Building skeleton-guided consistent multi-turn dialogues for large language models from scratch. In Proceedings of the 2025 Conference on Empirical Methods… find
  9. Sai Chen, Fengran Mo, Yanhao Wang, Cen Chen, Jian-Yun Nie, Chengyu Wang, and Jamie Cui. 2023. A customized text sanitization mechanism with differential privacy. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5747– 5758, Toronto, Canada. Association for Computational L… find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  11. Minxin Du, Xiang Yue, Sherman SM Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. 2023. Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2665– 2679. find
  12. Cynthia Dwork. 2006. Differential privacy. In International colloquium on automata, languages, and programming, pages 1–12. Springer. find
  13. Abbas Ghaddar, David Alfonso-Hermelo, Philippe Langlais, Mehdi Rezagholizadeh, Boxing Chen, and Prasanna Parthasarathi. 2024. CHARP: Conversation history AwaReness probing for knowledge-grounded dialogue systems. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1534– 155… find
  14. Xiaoyang Hou, Jian Liu, Jingyu Li, Yuhan Li, Wen-jie Lu, Cheng Hong, and Kui Ren. 2023. Ciphergpt: Secure two-party gpt inference. Cryptology ePrint Archive. find
  15. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697–10707. PMLR.
  16. Jonathan Katz and Yehuda Lindell. 2007. Introduction to modern cryptography: principles and protocols. Chapman and hall/CRC. find
  17. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  18. Peihua Mai, Ran Yan, Zhe Huang, Youjia Yang, and Yan Pang. 2024. Split-and-denoise: Protect large language model inference with local differential privacy. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 34281–343… find
  19. Frank McSherry and Kunal Talwar. 2007. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07), pages 94–103. IEEE. find
  20. Richard Plant, Dimitra Gkatzia, and Valerio Giuffrida. 2021. CAPE: Context-aware private embeddings for private language learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7970–7978, Online and Punta Cana, Dominican Republic. Association for Co… find
  21. Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training. find
  22. Sofya Raskhodnikova, Adam Smith, Homin K Lee, Kobbi Nissim, and Shiva Prasad Kasiviswanathan. 2008. What can we learn privately. In Proceedings of the 54th Annual Symposium on Foundations of Computer Science, pages 531–540. find
  23. Xicong Shen, Yang Liu, Yi Liu, Peiran Wang, Huiqi Liu, Jue Hong, Bing Duan, Zirui Huang, Yunlong Mao, Ye Wu, and Sheng Zhong. 2025. Sap: Privacy-preserving fine-tuning on language models with split-and-privatize framework. In International Joint Conference on Artificial Intelligence. 2024. Locally d… find
  24. Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Yuran Wang, Yong Ding, Yibo Zhang, Qi Zhang, and Xuan-Jing Huang. 2023. Textobfuscator: Making pre-trained language model a privacy protector via obfuscating word representations. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5459… find
  25. Congzheng Song and Ananth Raghunathan. 2020. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 377– 390. find
  26. Haotian Sun, Yuchen Zhuang, Wei Wei, Chao Zhang, and Bo Dai. 2024. Bbox-adapter: Lightweight adapting for black-box large language models. In International Conference on Machine Learning. find
  27. Meng Tong, Kejiang Chen, Jie Zhang, Yuang Qi, Weiming Zhang, Nenghai Yu, Tianwei Zhang, and Zhikun Zhang. 2025. Inferdpt: Privacy-preserving inference for closed-box large language models. IEEE Transactions on Dependable and Secure Computing, 22(5):4625–4640. find
  28. Jun Yan, Yijun Zhang, Laifeng Lu, Yi Tian, and Yihui Zhou. 2025. A graph generating method based on local differential privacy for preserving link relationships of social networks. Journal of Networking and Network Applications, 4(4):145–156. find
  29. Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, and Sherman S. M. Chow. 2021. Differential privacy for text analytics via natural text sanitization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3853–3866, Online. Association for Computational Linguis… find
  30. Tao Zhang, Chao Zhang, Feiyang Yuan, Lele Zheng, and Yiyun Guo. 2025. Alternating aggregation low-rank adaptation approach for federated large models. In International Conference on Advanced Data Mining and Applications, pages 418–425. Springer. find
  31. Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. 2018. Modeling multi-turn conversation with deep utterance aggregation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3740–3752, Santa Fe, New Mexico, USA. Association for Computational…
  32. Chunyan Zheng, Keke Sun, Wenhao Zhao, Haibo Zhou, Lixing Jiang, Shaoyang Song, and Chunlai Zhou. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-dp-3-differentially-private-prompt/