Elamur Research⌕
☰
home / papers / 2026 / Controlling What You Share: Assessing Language Model Adherence to Privacy Preferences

Controlling What You Share: Assessing Language Model Adherence to Privacy Preferences

Paper · 2026 · Guillem Ramírez, Alexandra Birch and Ivan Titov

Summary

Most people reach large language models through commercial APIs, so their personal data goes to the provider. People also differ in what they consider sensitive. The authors let users state their own privacy rules as free-form natural-language "privacy profiles." In their pipeline, a trusted local model first rewrites the query to remove protected details. A verifier then checks that the rewrite follows the profile. If it does, the rewrite goes to a stronger external model, and the local model combines that reply into the final answer. If not, the local model answers the original query by itself.

Two resources support this work. PEEP is a multilingual dataset of real WildChat queries, annotated for private content and paired with synthetic profiles. PROFIT is a training recipe: it runs the full pipeline, keeps only module outputs that are both useful and privacy-compliant, and fine-tunes each module on them with LoRA.

In few-shot use, only the 70B local model handled the privacy–quality trade-off well. After PROFIT, models as small as 3B matched or beat it. This makes user-controlled privacy feasible on small, on-device models.

Key points

  • PEEP dataset: 15,282 real user queries from WildChat. It covers private information beyond standard PII categories, and each query has a synthetic privacy profile. It is released on Hugging Face as guillemram97/PEEP.
  • PROFIT: trains the paraphraser, verifier and aggregator separately. Each module learns only from pipeline outputs that an LLM judge rated as good answers with low leakage of protected attributes.
  • Ablation: fine-tuning Llama 8B and 3B only on the external model's answers gave success rates of 0.512 and 0.465. Both are clearly below the full PROFIT pipeline, so the gains do not come merely from better final answers.
  • Generalisation to unseen tasks (GSM8K and Tatoeba en→eu, with GPT-4o-mini as the external model):
    • On GSM8K, Llama 8B with PROFIT scored 0.931, versus 0.829 for few-shot 8B and 0.927 for 70B.
    • On Tatoeba, it reached a COMET of 0.857, versus 0.778 and 0.844.
  • Robustness:
    • When phone number, religion, passport/ID and "has children" were held out during training, leakage of these attributes stayed low and was mostly lower than when they were included.
    • Results also held for profiles built from a survey of real user preferences.
    • The authors' own limitations: multi-turn dialogue is not covered, and the main profiles are synthetic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Guillem Ramírez, Alexandra Birch and Ivan Titov. 2026. Controlling What You Share: Assessing Language Model Adherence to Privacy Preferences. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1990

@inproceedings{ramirez2026controlling,
  title = {{Controlling What You Share: Assessing Language Model Adherence to Privacy Preferences}},
  author = {Guillem Ramírez, Alexandra Birch and Ivan Titov},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1990},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1990}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium, USENIX Security…
  2. Yu Chen, Tingxin Li, Huiming Liu, and Yang Yu. 2023. Hide and seek (has): A lightweight framework for prompt privacy protection. CoRR, abs/2309.03057. — [arXiv:2309.03057](https://arxiv.org/abs/2309.03057) find
  3. Amrita Roy Chowdhury, David Glukhov, Divyam Anshumaan, Prasad Chalasani, Nicolas Papernot, Somesh Jha, and Mihir Bellare. 2025. Prϵϵmpt: Sanitizing sensitive prompts for llms. Preprint, arXiv:2504.05147. — [arXiv:2504.05147](https://arxiv.org/abs/2504.05147) find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.
  5. Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid LLM: cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria,… find
  6. Cynthia Dwork. 2006. Differential privacy. In Automata, Languages and Programming, pages 1–12, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  7. Sahra Ghalebikesabi, Eugene Bagdasarian, Ren Yi, Itay Yona, Ilia Shumailov, Aneesh Pappu, Chongyang Shi, Laura Weidinger, Robert Stanforth, Leonard Berrada, Pushmeet Kohli, Po-Sen Huang, and Borja Balle. 2025. Privacy awareness for information-sharing assistants: A case-study on form-filling with co… find
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  9. Tommaso Green, Martin Gubri, Haritz Puerto, Sangdoo Yun, and Seong Joon Oh. 2025. Leaky thoughts: Large reasoning models are not private thinkers. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26507–26529, Suzhou, China. Association for Computationa… find
  10. Florian Hartmann, Duc-Hieu Tran, Peter Kairouz, Victor Cărbune, and Blaise Aguera Y Arcas. 2024. Can LLMs get help from other LLMs without revealing private information? In Proceedings of the Fifth Workshop on Privacy in Natural Language Processing, pages 107– 122, Bangkok, Thailand. Association fo… find
  11. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  12. Xianzhi Li, Ran Zmigrod, Zhiqiang Ma, Xiaomo Liu, and Xiaodan Zhu. 2024. Fine-tuning language models with differential privacy through adaptive noise allocation. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 8368–8375. Asso… find
  13. Sam Lin, Wenyue Hua, Zhenting Wang, Mingyu Jin, Lizhou Fan, and Yongfeng Zhang. 2025. EmojiPrompt: Generative prompt obfuscation for privacy-preserving communication with cloud-based LLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computation… find
  14. Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami Smaili, Rahul Gupta, and Richard S. Zemel. 2022. Differentially private decoding in large language models. CoRR, abs/2205.13621. find
  15. Omri Mendels, Coby Peled, Nava Vaisman Levy, Sharon Hart, Tomer Rosenthal, Limor Lahiani, and 1 others. 2018. Microsoft Presidio: Context aware, pluggable and customizable pii anonymization service for text and images. find
  16. Niloofar Mireshghallah, Maria Antoniak, Yash More, Yejin Choi, and Golnoosh Farnadi. 2024. Trust no bot: Discovering personal disclosures in human-llm conversations in the wild. In The First Conference on Language Modeling. find
  17. Ivoline Ngong, Swanand Kadhe, Hao Wang, Keerthiram Murugesan, Justin D. Weisz, Amit Dhurandhar, and Karthikeyan Natesan Ramamurthy. 2025. Protecting users from themselves: Safeguarding contextual privacy in interactions with conversational agents. Socially Responsible Language Modelling Research (So… find
  18. Helen Nissenbaum. 2009. Privacy in Context: Technology, Policy, and the Integrity of Social Life. Stanford University Press, USA. find
  19. Guillem Ramírez, Alexandra Birch, and Ivan Titov. 2024a. Optimising calls to large language models with uncertainty-based two-tier selection. The First Conference on Language Modeling, abs/2405.02134. — [arXiv:2405.02134](https://arxiv.org/abs/2405.02134) find
  20. Guillem Ramírez, Matthias Lindemann, Alexandra Birch, and Ivan Titov. 2024b. Cache & distil: Optimising API calls to large language models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 11838–11853. Associatio… find
  21. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. 62nd Annual Meeting of…
  22. Jiangou Zhan, Wenhui Zhang, Zheng Zhang, Huanran Xue, Yao Zhang, and Ye Wu. 2025. Portcullis: A scalable and verifiable privacy gateway for third-party LLM inference. In Thirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intell… find
  23. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations.
  24. Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2024. Universalner: Targeted distillation from large language models for open named entity recognition. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. find
  25. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. Green AI. Commun. ACM, 63(12):54– 63.
  26. Weiyan Shi, Ryan Shea, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. 2022. Just fine-tune twice: Selective differential privacy for large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, Decembe… find
  27. Li Siyan, Vethavikashini Chithrra Raghuram, Omar Khattab, Julia Hirschberg, and Zhou Yu. 2025. Papillon: Privacy preservation from internet-based and local language model ensembles. Proceedings of the North American Chapter of the Association for Computational Linguistics. find
  28. Rushil Thareja, Preslav Nakov, Praneeth Vepakomma, and Nils Lukas. 2025. Dp-fusion: Token-level differentially private inference for large language models. Preprint, arXiv:2507.04531. — [arXiv:2507.04531](https://arxiv.org/abs/2507.04531) find
  29. Jörg Tiedemann. 2020. The tatoeba translation challenge – realistic data sets for low resource and multilingual MT. In Proceedings of the Fifth Conference on Machine Translation, pages 1174–1182, Online. Association for Computational Linguistics.
  30. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large language models are not fair evaluators. In Proceedings of the find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ramirez-2026-controlling-what-you-share-assessing/