Elamur Research⌕
☰
home / papers / 2026 / PII-Bench: Evaluating Query-Aware Privacy Protection Systems

PII-Bench: Evaluating Query-Aware Privacy Protection Systems

Paper · 2026 · Hao Shen, Zhouhong Gu, Haokai Hong et al.

Summary

When people send prompts to LLM services, they often include personal details that could be intercepted or reused for training. Hiding every piece of PII protects the user but can leave the model unable to help. The authors propose a middle path: mask only the PII the query does not need. To evaluate this, they built PII-Bench, an automatically synthesized benchmark. It combines rule-based and LLM-based entity generation, consistency checks and generated user descriptions, covering cases from one person to many interacting people. Each sample pairs a query and a context with gold labels for query-relevant PII. The benchmark defines three tasks: PII Detection, Query-Related PII Detection and Query-Unrelated PII Masking. Experiments show that models, from BiLSTM-CRF to recent LLMs, find PII reasonably well. However, they struggle to judge which items the query actually needs, especially when many subjects appear. Small models lag further behind. The benchmark therefore isolates relevance reasoning as the main open problem for practical privacy filters in front of LLMs.

Key points

  • PII-Bench has 2,842 samples spanning 7 PII types and 55 fine-grained subcategories. A small real-data check set, PII-Real, contains 100 instances built from public profiles of 20 AI researchers and showed results consistent with the synthetic data.
  • The three masking strategies scored as follows (privacy / utility / balanced):
    • Query-unrelated masking: 0.83 / 0.89 / 0.86
    • Masking all PII: 1.00 / 0.52 / 0.76
    • No masking: 0.00 / 1.00 / 0.50
  • Accuracy falls sharply once a text has more than 5 subjects (F1 from 0.85 to 0.52), more than 33 PII entities or more than 3000 characters. On Test-Hard, the best approach (Self-Consistency) reaches only 0.463 F1.
  • On PII-Single, GPT-5, DeepSeek-V3.2 and DeepSeek-R1 achieve roughly 0.86–0.88 detection F1 but only about 0.73–0.77 on query relevance. The best system remains about 18 F1 points below humans.
  • Limitations include no coverage of medical or financial domains, no evaluation of inference attacks on masked text, and no assessment of the models' reasoning process.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

Cite

Cite as Hao Shen, Zhouhong Gu, Haokai Hong, Weili Han and Hongfeng Chai. 2026. PII-Bench: Evaluating Query-Aware Privacy Protection Systems. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.227

@inproceedings{shen2026pii,
  title = {{PII-Bench: Evaluating Query-Aware Privacy Protection Systems}},
  author = {Hao Shen, Zhouhong Gu, Haokai Hong, Weili Han and Hongfeng Chai},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.227},
  url = {https://doi.org/10.18653/v1/2026.acl-long.227}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Anthropic. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet. ↗
  3. Dimitris Asimopoulos, Ilias Siniosoglou, Vasileios Argyriou, Thomai Karamitsou, Eleftherios Foun-toukidis, Sotirios K Goudos, Ioannis D Moscholios, Konstantinos E Psannis, and Panagiotis Sarigiannidis. 2024. Benchmarking advanced text anonymisation methods: A comparative study on novel and tradition… — [arXiv:2404.14465](https://arxiv.org/abs/2404.14465) find
  4. David Biesner, Rajkumar Ramamurthy, Robin Stenzel, Max Lübbering, Lars Hillebrand, Anna Ladi, Maren Pielka, Rüdiger Loitz, Christian Bauckhage, and Rafet Sifa. 2022. Anonymization of german financial documents using neural network-based language models with contextual word representations. Internati… find
  5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  6. Tao Chen, Ruifeng Xu, Yulan He, and Xuan Wang. 2017. Improving sentiment analysis via sentence type classification using bilstm-crf and cnn. Expert Systems with Applications, 72:221–230. find
  7. Louise Deleger, Katalin Molnar, Guergana Savova, Fei Xia, Todd Lingren, Qi Li, Keith Marsolo, Anil Jegga, Megan Kaiser, Laura Stoutenborough, et al. 2013. Large-scale evaluation of automated clinical note de-identification and its impact on information extraction. Journal of the American Medical Inf… find
  8. Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. 2017. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association, 24(3):596–606. find
  9. Josep Domingo-Ferrer, David Sánchez, and Jordi Soria-Comas. 2022. Database anonymization: privacy models, data utility, and microaggregation-based inter-model connections. Springer Nature. find
  10. MM Douglass, GD Cliffford, Andrew Reisner, WJ Long, GB Moody, and RG Mark. 2005. De-identification algorithm for free-text nursing notes. In Computers in Cardiology, 2005, pages 331–334. IEEE. find
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  12. Mark Elliot, Elaine Mackey, Kieron O’Hara, and Caroline Tudor. 2016. The anonymisation decision-making framework. UKAN. find
  13. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
  14. Alistair EW Johnson, Lucas Bulgarelli, and Tom J Pollard. 2020. Deidentification of free-text medical records using pre-trained bidirectional transformers. In Proceedings of the ACM Conference on Health, Inference, and Learning, pages 214–221. find
  15. Qinbin Li, Junyuan Hong, Chulin Xie, Jeffrey Tan, Rachel Xin, Junyi Hou, Xavier Yin, Zhun Wang, Dan Hendrycks, Zhangyang Wang, et al. 2024. Llm-pbe: Assessing data privacy in large language models. arXiv preprint arXiv:2408.12787. find
  16. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  17. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023. Trustworthy llms: A survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374.
  18. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. Analyzing leakage of personally identifiable information in language models. In 2023 IEEE Symposium on Security and Privacy (SP), pages 346–363. IEEE. find
  19. Stephen Meisenbacher and Florian Matthes. 2024. Just rewrite it again: A post-processing method for enhanced semantic similarity and privacy preservation of differentially private rewritten text. arXiv preprint arXiv:2405.19831. — [arXiv:2405.19831](https://arxiv.org/abs/2405.19831) find
  20. Multi-Linguality Multi-Functionality Multi-Granularity. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.
  21. Yuta Nakamura, Shouhei Hanaoka, Yukihiro Nomura, Naoto Hayashi, Osamu Abe, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2020. Kart: Privacy leakage framework of language models pre-trained with clinical records. arXiv preprint arXiv:2101.00036. — [arXiv:2101.00036](https://arxiv.org/abs/2101.00036) find
  22. OpenAI. 2024. Hello gpt-4o. https://openai.com/ index/hello-gpt-4o. ↗
  23. Anthi Papadopoulou, Yunhao Yu, Pierre Lison, and Lilja Øvrelid. 2022. Neural text sanitization with explicit measures of privacy risk. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natu… find
  24. Fatemeh Khoda Parast, Chandni Sindhav, Seema Nikam, Hadiseh Izadi Yekta, Kenneth B Kent, and Saqib Hakak. 2022. Cloud computing security: A survey of service-based models. Computers & Security, 114:102580. find
  25. Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet. 2022. The text anonymization benchmark (tab): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics, 48(4):1053–1101.
  26. Patrick Ruch, Robert H Baud, Anne-Marie Rassinoux, Pierrette Bouillon, and Gilbert Robert. 2000. Medical document anonymization with a semantic lexicon. In Proceedings of the AMIA Symposium, page 729. American Medical Informatics Association. find
  27. Zhili Shen, Zihang Xi, Ying He, Wei Tong, Jingyu Hua, and Sheng Zhong. 2024. The fire thief is also the keeper: Balancing usability and privacy in prompts. arXiv preprint arXiv:2406.14318. — [arXiv:2406.14318](https://arxiv.org/abs/2406.14318) find
  28. Shreya Singhal, Andres Felipe Zambrano, Maciej Pankiewicz, Xiner Liu, Chelsea Porter, and Ryan S Baker. 2024. De-identifying student personally identifying information with gpt-4. In Proceedings of the 17th International Conference on Educational Data Mining, pages 559–565. find
  29. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-pii-bench-evaluating-query-aware/