Elamur Research⌕
☰
home / papers / 2026 / The Potential for Misleading Results in Text Sanitisation with Standard Evaluation Metrics

The Potential for Misleading Results in Text Sanitisation with Standard Evaluation Metrics

Summary

Evaluating systems that detect and mask personal identifiable information (PII) in free text is difficult. Open data is scarce, so researchers often rely on Wikipedia biographies, where identifying content is very dense. Zhang and Anderson show that when the target class is this prevalent, weak or near-random detectors can look strong under precision, recall and F1.

As a case study, they re-examine the word-embedding sanitisation method of Hassan et al. (2023). That method flags noun phrases whose cosine similarity to a protected entity exceeds a threshold. The authors find that its apparent advantage over NER baselines comes largely from the evaluation design. Limiting candidates to noun phrases raises the share of sensitive items, and the similarity scores barely separate a document's own phrases from phrases taken from other documents.

They then compare many binary classification metrics in high-prevalence settings. Most do not reveal predictions that simply follow the class prevalence, but the Matthews Correlation Coefficient (MCC) does. The paper offers practical guidance for benchmarking text sanitisation, especially on biography-style corpora.

Key points

  • The case study uses a Word2Vec model trained on abstracts of 19,000 Wikipedia articles about movie actors. It is evaluated on 50 manually annotated abstracts and compared with Stanford NER3, NER4, NER7 and Microsoft Presidio.
  • The original results showed lower precision but much higher recall and F1 than the NER tools. On closer inspection, cosine scores within a document were only marginally above cross-document scores, and longer noun phrases tended to score higher.
  • Restricting evaluation to noun phrases is a reasonable pre-processing step, but it artificially increases the prevalence of sensitive spans. The more accurate this heuristic is, the worse the metric distortion becomes.
  • Among the metrics compared, MCC was clearly the most reliable at exposing prevalence-driven performance. F1 tended to overstate quality and could change system rankings, even on more balanced data (shown in Table 10).
  • The authors recommend reporting MCC at least alongside other metrics. They also call for MCC-based or complementary measures tailored to the privacy–utility trade-off in sanitisation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4ubbuzpc4hpu

@inproceedings{zhang2026potential,
  title = {{The Potential for Misleading Results in Text Sanitisation with Standard Evaluation Metrics}},
  author = {Zhang, Dan and Anderson, Mark},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4ubbuzpc4hpu},
  url = {https://doi.org/10.63317/4ubbuzpc4hpu}
}

References

22 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hadeer Ahmed, Issa Traore, Sherif Saad, and Mohammad Mamun. 2021. Automated detection of unstructured context-dependent sensitive information using deep learning. Internet of Things, 16:100444. find
  2. Federico Albanese, Daniel Ciolek, and Nicolas D’Ippolito. 2023. Text sanitization beyond specific domains: Zero-shot redaction & substitution with large language models. arXiv preprint arXiv:2311.10785. — [arXiv:2311.10785](https://arxiv.org/abs/2311.10785) find
  3. Davide Chicco and Giuseppe Jurman. 2023. The matthews correlation coefficient (mcc) should replace the roc auc as the standard metric for assessing binary classification. BioData Mining, 16(1):4. find
  4. Louise Deleger, Katalin Molnar, Guergana Savova, Fei Xia, Todd Lingren, Qi Li, Keith Marsolo, Anil Jegga, Megan Kaiser, Laura Stoutenborough, et al. 2013. Large-scale evaluation of automated clinical note de-identification and its impact on information extraction. Journal of the American Medical Inf… find
  5. Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. 2017. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association, 24(3):596–606. find
  6. Longjam V. Devi and N. Saharia. 2025. Detection of sensitive information using crf in unstructured text. In Recent Advances in NLP, chapter 18. CRC Press. find
  7. Chongomweru Halimu, Asem Kasem, and SH Shah Newaz. 2019. Empirical comparison of area under roc curve (auc) and mathew correlation coefficient (mcc) for evaluating machine learning algorithms on imbalanced datasets for binary classification. In Proceedings of the 3rd international conference on mach… find
  8. Fadi Hassan, David Sánchez, and Josep Domingo-Ferrer. 2023. Utility-preserving privacy protection of textual documents via word embeddings. IEEE Transactions on Knowledge and Data Engineering, 35(1):1058–1071. find
  9. Celestine Iwendi, Syed Atif Moqurrab, Adeel Anjum, Sangeen Khan, Senthilkumar Mohan, and Gautam Srivastava. 2020. N-sanitization: A semantic privacy-preserving framework for unstructured medical datasets. Computer Communications, 161:160–171. find
  10. Kristian Nørgaard Jensen, Mike Zhang, and Barbara Plank. 2021. De-identification of privacy-related entities in job postings. find
  11. Bennett Kleinberg, Toby Davies, and Maximilian Mozes. 2022. Textwash–automated open-source text anonymisation. arXiv preprint arXiv:2208.13081. — [arXiv:2208.13081](https://arxiv.org/abs/2208.13081) find
  12. Zengjian Liu, Buzhou Tang, Xiaolong Wang, and Qingcai Chen. 2017. De-identification of clinical notes via recurrent neural network and conditional random field. Journal of biomedical informatics, 75:S34–S42. find
  13. Zhengliang Liu, Yue Huang, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Yiwei Li, Peng Shu, et al. 2023. Deid-gpt: Zero-shot medical text de-identification by gpt-4. arXiv preprint arXiv:2303.11032. — [arXiv:2303.11032](https://arxiv.org/abs/2303.11032) find
  14. Isaac Martín De Diego, Ana R Redondo, Rubén R Fernández, Jorge Navarro, and Javier M Moguerza. 2021. General performance score for classification problems. find
  15. Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451. find
  16. Ishna Neamatullah, Margaret M Douglass, Li-Wei H Lehman, Andrew Reisner, Mauricio Villarroel, William J Long, Peter Szolovits, George B Moody, Roger G Mark, and Gari D Clifford. 2008. Automated de-identification of free-text medical records. BMC medical informatics and decision making, 8(1):32. find
  17. Annika Willoch Olstad, Anthi Papadopoulou, and Pierre Lison. 2023. Generation of replacement options in text sanitization. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 292–300, Tórshavn, Faroe Islands. University of Tartu Library. find
  18. Anthi Papadopoulou, Pierre Lison, Lilja Øvrelid, and Ildikó Pilán. 2022a. Bootstrapping text anonymization models with distant supervision. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4477–4487, Marseille, France. European Language Resources Association. find
  19. Anthi Papadopoulou, Yunhao Yu, Pierre Lison, and Lilja Øvrelid. 2022b. Neural text sanitization with explicit measures of privacy risk. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Nat… find
  20. Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet. 2022. The text anonymization benchmark (tab): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics, 48(4):1053–1101. find
  21. Emily M Weitzenboeck, Pierre Lison, Malgorzata Cyndecka, and Malcolm Langford. 2022. The gdpr and unstructured data: is anonymization possible? International Data Privacy Law, 12(3):184–206. find
  22. Hui Yang and Jonathan M Garibaldi. 2015. Automatic detection of protected health information from clinic narratives. Journal of biomedical informatics, 58:S30–S38. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-potential-misleading-results-text-sanitisation/