Elamur Research⌕
☰
home / papers / 2026 / Subject-level Inference for Realistic Text Anonymization Evaluation

Subject-level Inference for Realistic Text Anonymization Evaluation

Paper · 2026 · Myeong Seok Oh, Dong-Yun Kim, Hanseok Oh et al.

Summary

Text anonymization is usually judged by how many PII spans get masked. That approach misses two risks. A capable LLM adversary can often reconstruct hidden details from context. And documents frequently describe several people, not only one target. The authors introduce SPIA (Subject-level PII Inference Assessment), a benchmark that scores each person mentioned in a text rather than each span.

SPIA combines 144 ECHR judgments from TAB with 531 synthetic online texts from PANORAMA. The resulting 675 documents contain 1,712 subjects and 7,040 PII annotations. Evaluation runs in two stages: first every identifiable subject is located, then PII is inferred separately for each one. The results are summarized with two newly proposed metrics, IPR (Individual Protection Rate) and CPR (Collective Protection Rate).

The authors tested 4 anonymization methods with 6 LLM backbones. Even when masking covered more than 90% of PII spans, inference-based protection could fall to about 33%. Methods that focus on one target person left the other people in the document noticeably more exposed. Results also differed by domain. The work argues that subject-aware, inference-based evaluation is needed before anonymized text can be considered safe.

Key points

  • Dataset and categories: SPIA has 675 English documents (TAB 144, PANORAMA 531). Its 15 PII categories split into 5 CODE types with fixed formats (e.g., phones, emails) and 10 NON-CODE free-text types (e.g., names, age).
  • Annotation: 295 documents were labeled fully by humans. Another 380 were pre-labeled by an LLM and then reviewed by people. On a cross-labeled 20% subset, subject match exceeded 95% and PII agreement exceeded 91%.
  • Pipeline validation: 11 LLMs were checked against the human labels. Claude-Sonnet-4.5 performed best, with about 96% subject match and about 91% PII accuracy.
  • Comparison with prior benchmarks: The authors claim SPIA is the first benchmark to offer all four properties below; TAB, PersonalReddit, PII-Bench, PANORAMA and others each cover only some of them.
    • broad PII coverage
    • inference-based evaluation
    • multiple domains
    • per-subject scoring
  • Availability and limitations: Code and data are released at github.com/maisonOP/spia.
    • The IPR and CPR metrics give every PII category equal weight.
    • Only English texts are covered.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada54 works · 13,565 citations🎓Yonsei UniversitySeoul · South Korea17 works · 501 citations🎓Soongsil UniversitySeoul · South Korea2 works · 7 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Myeong Seok Oh, Dong-Yun Kim, Hanseok Oh, Chaean Kang, Joeun Kang, Xiaonan Wang, Hyunjung Park, Young Cheol Jung and Hansaem Kim. 2026. Subject-level Inference for Realistic Text Anonymization Evaluation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.778

@inproceedings{oh2026subject,
  title = {{Subject-level Inference for Realistic Text Anonymization Evaluation}},
  author = {Myeong Seok Oh, Dong-Yun Kim, Hanseok Oh, Chaean Kang, Joeun Kang, Xiaonan Wang, Hyunjung Park, Young Cheol Jung and Hansaem Kim},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.778},
  url = {https://doi.org/10.18653/v1/2026.acl-long.778}
}

References

43 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Introducing Claude Sonnet 4.5. find
  2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
  3. Mirco Beltrame, Mauro Conti, Pierpaolo Guglielmin, Francesco Marchiori, and Gabriele Orazi. 2024. RedactBuster: Entity type recognition from redacted documents. In Computer Security – ESORICS 2024, volume 14983 of Lecture Notes in Computer Science, pages 451–470. Springer. find
  4. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In Proceedings of the 30th USENIX Security Symposium,…
  5. Yizhuo Chen, Chun-Fu Chen, Hsiang Hsu, Shaohan Hu, Marco Pistoia, and Tarek F. Abdelzaher. 2024. MaSS: Multi-attribute selective suppression for utility-preserving data transformation from an information-theoretic perspective. In Proceedings of the 41st International Conference on Machine Learning,… find
  6. Tobias Deußer, Lorenz Sparrenberg, Armin Berger, Max Hahnbück, Christian Bauckhage, and Rafet Sifa. 2025. A survey on current trends and recent advances in text anonymization. In Proceedings of the IEEE International Conference on Big Data. IEEE. find
  7. Josep Domingo-Ferrer, David Sánchez, and Jordi Soria-Comas. 2016. Database Anonymization: Privacy Models, Data Utility, and Microaggregation-based Inter-model Connections. Synthesis Lectures on Information Security, Privacy, and Trust. Springer International Publishing, Cham. find
  8. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography, pages 265–284, Berlin, Heidelberg. Springer. find
  9. Mark Elliot, Elaine Mackey, and Kieron O’Hara. 2020. The Anonymisation Decision-Making Framework, 2nd edition. UKAN. find
  10. European Union. 2016. General data protection regulation (GDPR). find
  11. Li Fei, Yejee Kang, Seoyoon Park, Yeonji Jang, Jongkyu Lee, and Hansaem Kim. 2024. KDPII: A new korean dialogic dataset for the deidentification of personally identifiable information. IEEE Access, 12:135626– 135641. find
  12. Ahmed Frikha, Nassim Walha, Krishna Kanth Nakka, Ricardo Mendes, Xue Jiang, and Xuebing Zhou. 2024. IncogniText: Privacy-enhancing conditional text anonymization via LLM-based private attribute randomization. In NeurIPS 2024 Workshop on Safe Generative AI.
  13. Gemma Team. 2025. Gemma 3 technical report. Computing Research Repository, arXiv:2503.19786. information in language models. In 2023 IEEE Symposium on Security and Privacy (SP), pages 346–363. IEEE. find
  14. Benet Manzanares-Salor, David Sánchez, and Pierre Lison. 2024. Evaluating the disclosure risk of anonymized documents via a machine learning-based re-identification attack. Data Mining and Knowledge Discovery, 38(6):4040–4075. find
  15. Philippe Golle. 2006. Revisiting the uniqueness of simple demographics in the US population. In Proceedings of the 5th ACM Workshop on Privacy in Electronic Society, WPES ’06, pages 77–80, New York, NY, USA. Association for Computing Machinery. find
  16. Mariana Monteiro, Filipe Correia, Paulo Queiroz, Rui Ramos, Dinis Trigo, and Gonçalo Gonçalves. 2024. Patterns of data anonymization. In Proceedings of the 29th European Conference on Pattern Languages of Programs, People, and Practices, EuroPLoP ’24, pages 1–9, New York, NY, USA. Association for Co… find
  17. Aaron Grattafiori and 1 others. 2024. The Llama 3 herd of models. Computing Research Repository, arXiv:2407.21783.
  18. OpenAI. 2025a. gpt-oss-120b & gpt-oss-20b model card. Computing Research Repository, arXiv:2508.10925.
  19. Rajitha Hathurusinghe, Isar Nejadgholi, and Miodrag Bolic. 2021. A privacy-preserving approach to extraction of personal information through automatic annotation and federated learning. In Proceedings of the Third Workshop on Privacy in Natural Language Processing, pages 36–45, Online. Association f… find
  20. OpenAI. 2025b. Introducing GPT-4.1 in the API. find
  21. Neel Kanwal, Emiel A. M. Janssen, and Kjersti Engan. 2024. Balancing privacy and progress in artificial intelligence: Anonymization in histopathology for biomedical research and education. In Frontiers of Artificial Intelligence, Ethics, and Multidisciplinary Applications, pages 417–429, Singapore.… find
  22. Iyadh Ben Cheikh Larbi, Aljoscha Burchardt, and Roland Roller. 2022. Which anonymization technique is best for which NLP task? – it depends. a systematic study on clinical text processing. Computing Research Repository, arXiv:2209.00262. — [arXiv:2209.00262](https://arxiv.org/abs/2209.00262) find
  23. Pierre Lison, Ildikó Pilán, David Sanchez, Montserrat Batet, and Lilja Øvrelid. 2021. Anonymisation models for text data: State of the art, challenges and future directions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Co… find
  24. Zhengliang Liu, Yue Huang, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Yiwei Li, Peng Shu, Fang Zeng, Lichao Sun, Wei Liu, Dinggang Shen, Quanzheng Li, Tianming Liu, Dajiang Zhu, and Xiang Li. 2023. DeID-GPT: Zero-shot medical text de-identification by GPT-4. Computing Research… — [arXiv:2303.11032](https://arxiv.org/abs/2303.11032) find
  25. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. Analyzing leakage of personally identifiable Anthi Papadopoulou, Pierre Lison, Mark Anderson, Lilja Øvrelid, and Ildikó Pilán. 2023. Neural text sanitization with privacy risk indicators: An empi… — [arXiv:2310.14312](https://arxiv.org/abs/2310.14312) find
  26. Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet. 2022. The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics, 48(4):1053–1101.
  27. Ildikó Pilán, Benet Manzanares-Salor, David Sánchez, and Pierre Lison. 2025. Truthful text sanitization guided by inference attacks. Applied Soft Computing. find
  28. Qwen Team. 2025. Qwen3 technical report. Computing Research Repository, arXiv:2505.09388.
  29. Luc Rocher, Julien Hendrickx, and Yves-Alexandre de Montjoye. 2019. Estimating the success of re-identifications in incomplete datasets using generative models. Nature Communications, 10(1):3069. find
  30. Sriram Selvam and Anneswa Ghosh. 2025. PANORAMA: A synthetic PII-laced dataset for studying sensitive data memorization in LLMs. Computing Research Repository, arXiv:2505.12238. — [arXiv:2505.12238](https://arxiv.org/abs/2505.12238) find
  31. Hao Shen, Zhouhong Gu, Haokai Hong, and Weili Han. 2025. PII-Bench: Evaluating query-aware privacy protection systems. Computing Research Repository, arXiv:2502.18545.
  32. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy, pages 3–18.
  33. Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. 2024. Beyond memorization: Violating privacy via inference with large language models. In The Twelfth International Conference on Learning Representations, pages 33832–33878. find
  34. Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. 2025. Language models are advanced anonymizers. In The Thirteenth International Conference on Learning Representations, pages 98558– 98598. find
  35. Amber Stubbs and Özlem Uzuner. 2015. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus. Journal of Biomedical Informatics, 58(Suppl):S20–S29. find
  36. Latanya Sweeney. 2000. Simple demographics often identify people uniquely. Health (San Francisco), 671. find
  37. Latanya Sweeney. 2002. k-anonymity: A model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10(5):557– 570. find
  38. Type Category Total CODE ID Number Driver License Phone Passport Email 95 75 85 31 90 – – – – – 95 75 85 31 90 NON-CODE Name Sex Age Location Nationality Education Relationship Occupation Affiliation Position 691 556 159 452 452 101 254 441 193 15 366 392 153 510 519 294 55 453 292 316 1,057 948 312… find
  39. Saiteja Utpala, Sara Hooker, and Pin-Yu Chen. 2023. Locally differentially private document generation using zero shot prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8442–8457, Singapore. Association for Computational Linguistics. find
  40. Shang Wang, Tianqing Zhu, Bo Liu, Ming Ding, Dayong Ye, Wanlei Zhou, and Philip Yu. 2025a. Unique security and privacy threats of large language model: A comprehensive survey. ACM Computing Surveys, 58(4):83:1–83:36. find
  41. Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. 2025b. Generalization v.s. memorization: Tracing language models’ capabilities back to pretraining data. Computing Research Repository, arXiv:2407.14985. — [arXiv:2407.14985](https://arxiv.org/abs/2407.14985) find
  42. Tianyu Yang, Xiaodan Zhu, and Iryna Gurevych. 2025. Robust utility-preserving text anonymization based on large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28922–28941, Vienna, Austria. Association for Com… find
  43. Hanna Yukhymenko, Robin Staab, Mark Vero, and Martin Vechev. 2024. A synthetic dataset for personal attribute inference. Advances in Neural Information Processing Systems, 37:120735–120779. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oh-2026-subject-level-inference-realistic-text/