Elamur Research⌕
☰
home / papers / 2026 / The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques

The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques

Paper · 2026 · Sebastian Ochs and Ivan Habernal

Summary

Tools that strip personally identifiable information (PII) from text are widely used to share sensitive legal and medical documents. Several recent studies claim that attacks, often using LLMs, can recover the removed details. Ochs and Habernal argue that these reported success rates are inflated.

They re-examine earlier attack evaluations and find that data leakage and contamination are rarely controlled. The supposedly private texts are often already public, or the attack model may have memorized them during pretraining. A successful recovery may therefore show what the attacker already knew, not a weakness in the sanitization.

The authors consider which data sources and setups could avoid this problem. They also run two small experiments on material unlikely to be in pretraining corpora: Czech court announcements and English YouTube travel vlogs, scored with EM@3. The results suggest that some PII can still be inferred from sanitized text alone. However, a reliable verdict requires truly private data, which public researchers generally cannot obtain. The paper therefore questions the evidence behind claims about both privacy risk and protection, and it calls for formal threat models.

Key points

  • The paper asks two research questions. Are current evaluations of PII reconstruction attacks flawed? The answer is yes, because leakage is not excluded. Can public researchers fix this without sensitive data? The answer is not in an openly verifiable, reproducible way.
  • Among the attacks the paper critiques are those by Charpentier and Lison (2025), Lukas et al. (2023), Patsakis and Lykousas (2023), and Nyffenegger, Stürmer, and Niklaus (2024). Proprietary LLMs with undisclosed training data are singled out as especially hard to assess.
  • The authors' own experiments use Czech court announcements (legal domain) and personal English YouTube vlogs, with a single prompt and the EM@3 metric. Acknowledged limitations are possible contamination of the Czech data and reliance on one prompt.
  • The background contrasts GDPR with HIPAA's Safe Harbor method, which requires removing 18 PHI types. It also cites Weitzenboeck et al. (2022): under the strictest GDPR reading, redacted text becomes useless when the adversary holds the originals.
  • As a way forward, the authors propose formal threat models covering data access, attacker capabilities, and defenses. They judge DP, k-anonymity, and Contextual Integrity poorly suited to this problem, and they leave the formal framework itself to future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany114 works · 36,326 citations🎓Ruhr University BochumBochum · Germany27 works · 1,189 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 worksDecision Sciences · 249 worksManagement Science and Operations Research · 134 works

🧭 Topics 1Research subjects and languages the work is about.Legal AI · 150 works

Cite

Cite as Sebastian Ochs and Ivan Habernal. 2026. The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.a.615

@article{ochs2026conundrum,
  title = {{The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques}},
  author = {Sebastian Ochs and Ivan Habernal},
  journal = {Computational Linguistics},
  year = {2026},
  doi = {10.1162/coli.a.615},
  url = {https://doi.org/10.1162/coli.a.615}
}

References

40 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adrian, David, Karthikeyan Bhargavan, Zakir Durumeric, Pierrick Gaudry, Matthew Green, J. Alex Halderman, Nadia Heninger, Drew Springall, Emmanuel Thomé, Luke Valenta, et al. 2015. Imperfect Forward Secrecy: How Diffie-Hellman Fails in Practice. In Proceedings of the 22nd ACM SIGSAC Conference on C… find
  2. Aviram, Nimrod, Sebastian Schinzel, Juraj Somorovsky, Nadia Heninger, Maik Dankel, Jens Steube, Luke Valenta, David Adrian, J. Alex Halderman, Viktor Dukhovni, et al. 2016. DROWN: Breaking TLS using SSLv2. In Proceedings of the 25th USENIX Conference on Security Symposium, pages 689–706. Balloccu, S… — [doi:10.18653/v1/2024](https://doi.org/10.18653/v1/2024) find
  3. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901.
  4. Carlini, Nicholas, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations. https://doi.org/10.52202/075280-1708 Carlini, Nicholas, Florian… ↗
  5. Carrell, David S., Bradley A. Malin, David J. Cronkite, John S. Aberdeen, Cheryl Clark, Muqun Li, Dikshya Bastakoty, Steve Nyemba, and Lynette Hirschman. 2020. Resilience of clinical text de-identified with “hiding in plain sight” to hostile reidentification attacks by human readers. Journal of the…
  6. Costa, Paul T. and Robert R. McCrae. 2008. The Revised NEO Personality Inventory (NEO-PI-R). In The SAGE Handbook of Personality Theory and Assessment: Volume 2—Personality Measurement and Testing. SAGE Publications Ltd, pages 179–198. https://doi.org/10.4135/9781849200479.n9 Costa-jussà, Marta R.… — [doi:10.1038/s41586-024](https://doi.org/10.1038/s41586-024) ↗
  7. Diffie, W. and M. Hellman. 1976. New directions in cryptography. IEEE Transactions on Information Theory, 22(6):644–654. https://doi.org/10 .1109/TIT.1976.1055638 Dorr, D. A., W. F. Phillips, S. Phansalkar, S. A. Sims, and J. F. Hurdle. 2006. Assessing the difficulty and time cost of de-identificati… — [doi:10.1055/s-0038-1634080](https://doi.org/10.1055/s-0038-1634080) find
  8. Elazar, Yanai and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11–21. https://doi.org/10.18653/v1/D18-1002 ↗
  9. European Commission. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regu… find
  10. Ford, Elizabeth, John A. Carroll, Helen E. Smith, Donia Scott, and Jackie A. Cassell. 2016. Extracting information from the text of electronic medical records to improve case detection: A systematic review. Journal of the American Medical Informatics Association, 23(5):1007–1015. https://doi.org/10… — [doi:10.1093/jamia/ocv180](https://doi.org/10.1093/jamia/ocv180) find
  11. Holmes, Langdon, Scott Crossley, Harshvardhan Sikka, and Wesley Morris. 2023. PIILO: An open-source system for personally identifiable information labeling and obfuscation. Information and Learning Sciences, 124(9/10):266–284. https://doi.org/10.1108/ILS-04 -2023-0032 Huang, Baixiang, Canyu Chen, an… ↗
  12. Ippolito, Daphne, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette Choo, and Nicholas Carlini. 2023. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In Proceedings of the 16th International Natural La…
  13. Kifer, Daniel, John M. Abowd, Robert Ashmead, Ryan Cumings-Menon, Philip Leclerc, Ashwin Machanavajjhala, William Sexton, and Pavel Zhuravlev. 2022. find
  14. Bayesian and frequentist semantics for common variations of differential privacy: Applications to the 2020 Census. arXiv preprint arXiv:2209.03310. — [arXiv:2209.03310](https://arxiv.org/abs/2209.03310) find
  15. Kifer, Daniel and Ashwin Machanavajjhala. 2011. No free lunch in data privacy. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data (PODS 11), pages 193–204. https://doi.org/10.1145/1989323.1989345 Kim, Siwon, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and… — [doi:10.1145/1989323.1989345](https://doi.org/10.1145/1989323.1989345) ↗
  16. Kleinberg, Bennett, Toby Davies, and Maximilian Mozes. 2022. Textwash – automated open-source text anonymisation. arXiv:2208.13081. — [arXiv:2208.13081](https://arxiv.org/abs/2208.13081) find
  17. Klimt, Bryan and Yiming Yang. 2004. The Enron corpus: A new dataset for email classification research. In Machine Learning: ECML 2004, pages 217–226. https://doi.org/10.1007/978-3-540-30115-8 22 Kocetkov, Denis, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell,… ↗
  18. Leonard, Robert A., Juliane E. R. Ford, and Tanya Karoli Christensen. 2017. Forensic linguistics: Applying the science of linguistics to issues of the law. Hofstra Law Review, 45(3):Article 11. find
  19. Li, Haoran, Wei Fan, Yulin Chen, Cheng Jiayang, Tianshu Chu, Xuebing Zhou, Peizhao Hu, and Yangqiu Song. 2025. Privacy checklist: Privacy violation detection grounding on contextual integrity theory. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for…
  20. Liu, Zhengliang, Yue Huang, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Yiwei Li, Peng Shu, Fang Zeng, Lichao Sun, Wei Liu, Dinggang Shen, Quanzheng Li, Tianming Liu, Dajiang Zhu, and Xiang Li. 2023. DeID-GPT: Zero-shot medical text de-identification by GPT-4. arXiv:2303.11032 L…
  21. Meisenbacher, Stephen and Florian Matthes. 2024. Thinking outside of the differential privacy box: A case study in text privatization with language model prompting. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5656–5665. https://doi.org/10.18653/v1… ↗
  22. Google Security Advisory. Morris, John, Justin Chiu, Ramin Zabih, and Alexander Rush. 2022. Unsupervised text deidentification. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4777–4788. https://doi.org/10.18653/v1/2022.findings -emnlp.352 Mouhammad, Nina, Johannes Da… ↗
  23. Neamatullah, Ishna, Matthew M. Douglass, Li-Wei H. Lehman, Andrew T. Reisner, Mauricio Villarroel, William J. Long, Roger G. Mark, and Gari D. Clifford. 2008. Automated de-identification of free-text medical records. BMC Medical Informatics and Decision Making, 8(1):32. https://doi.org/10.1186/1472… ↗
  24. Olstad, Annika Willoch, Anthi Papadopoulou, and Pierre Lison. 2023. Generation of replacement options in text sanitization. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 292–300. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Appleba… find
  25. Pal, Anwesan, Radhika Bhargava, Kyle Hinsz, Jacques Esterhuizen, and Sudipta Bhattacharya. 2024. The empirical impact of data sanitization on language models. In Neurips Safe Generative AI Workshop 2024. find
  26. Papadopoulou, Anthi, Yunhao Yu, Pierre Lison, and Lilja Øvrelid. 2022. Neural text sanitization with explicit measures of privacy risk. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Nat… find
  27. Pillutla, Krishna, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In Proceedings of the 35th International Conference on Neural Information Processing Syste…
  28. Potter, Danielle, Raven Brothers, Andrej Kolacevski, Jacob E. Koskimaki, Amy McNutt, Robert S. Miller, Jatin Nagda, Anil Nair, Wendy S. Rubinstein, Andrew K. Stewart, Iris J. Trieb, and George A. Komatsoulis. 2020. Development of CancerLinQ, a health information learning platform from multiple elect… find
  29. Seegmiller, Parker and Sarah Preum. 2023. Statistical depth for ranking and characterizing transformer-based text embeddings. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9600–9611. https://doi.org/10.18653/v1/2023 .emnlp-main.596 Soldaini, Luca, R… ↗
  30. Staab, Robin, Mark Vero, Mislav Balunovic, and Martin Vechev. 2025. Language models are advanced anonymizers. In The Thirteenth International Conference on Learning Representations. find
  31. Stevens, Marc, Elie Bursztein, Pierre Karpman, Ange Albertini, and Yarik Markov. 2017. The first collision for full SHA-1. In Advances in Cryptology – CRYPTO 2017, pages 570–596. https://doi.org/10.1007/978-3-319 -63688-7 19 Stubbs, Amber and Özlem Uzuner. 2015. Annotating longitudinal clinical nar… ↗
  32. Sweeney, Latanya. 2000. Simple demographics often identify people uniquely. Health (San Francisco), 671(2000):1–34. find
  33. Sweeney, Latanya. 2002. k-anonymity: A model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10(5):557–570. https://doi.org/10.1142/S0218488502001648 Sweeney, Latanya, Ji Su Yoo, Laura J. Perovich, Katherine E. Boronow, Phil Brown, and Julia Gree… ↗
  34. Terzidou, Kalliopi. 2023. Automated anonymization of court decisions: Facilitating the publication of court decisions through algorithmic systems. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL ’23, pages 297–305. https://doi.org/10.1145/3594536… — [doi:10.1145/3594536](https://doi.org/10.1145/3594536) ↗
  35. Congress. 1996. Health Insurance Portability and Accountability Act of 1996. U.S. Public Law. Public Law 104-191. find
  36. U.S. Government. 2013. Code of Federal Regulations, Title 45, Section 164.514b(1) (2013). https://www.ecfr.gov/current/title-45/part-164/section-164 .514\#p-164.514(b) Utpala, Saiteja, Sara Hooker, and Pin-Yu Chen. 2023. Locally differentially private document generation using zero shot prompting. I… — [doi:10.18653/v1/2023.findings](https://doi.org/10.18653/v1/2023.findings) ↗
  37. Wei, Jason, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions on M…
  38. Survey Certification. Weitzenboeck, Emily M., Pierre Lison, Malgorzata Cyndecka, and Malcolm Langford. 2022. The GDPR and unstructured data: Is anonymization possible? International Data Privacy Law, 12(3):184–206. https://doi.org/10 .1093/idpl/ipac008 Xie, Chulin, Zinan Lin, Arturs Backurs, Sivakan… find
  39. Yang, Tianyu, Xiaodan Zhu, and Iryna Gurevych. 2025. Robust utility-preserving text anonymization based on large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28922–28941. https://doi.org/10.18653/v1/2025.a… ↗
  40. Zhang, Meng, Xiangyang Luo, and Ningbo Huang. 2025. Social media user geolocation based on large language models. In Data Security and Privacy Protection, pages 304–312. https://doi.org/10.1007/978-981-97-8540 -7 19 Zhong, Haoxi, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Su… — [doi:10.1007/978-981-97-8540](https://doi.org/10.1007/978-981-97-8540) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ochs-2026-conundrum-trustworthy-research-attacking-personally/