Elamur Research⌕
☰
home / papers / 2022 / The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization

The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization

Paper · 2022 · Ildikó Pilán, Pierre Lison, Lilja Øvrelid et al.

Summary

Most privacy-related NLP resources target de-identification, which removes only predefined categories such as names or phone numbers. Real anonymization, as GDPR understands it, also requires hiding indirect clues that could reveal a person's identity in combination. Because suitable annotated data was missing, it was hard to judge how well existing tools actually protect individuals. The authors address this with TAB, an open English corpus of European Court of Human Rights (ECHR) judgments. Annotators treated each document as a genuine anonymization exercise and decided which spans must be masked to shield the person in question. Court cases suit this purpose because they contain rich, open-ended biographical detail. Alongside the data, the paper proposes new evaluation measures. Privacy is scored per entity rather than per mention and weights direct identifiers more heavily. Utility is estimated from how informative the masked terms are. Tests with several baselines show that the task is difficult and that standard NER-style de-identification falls short. TAB therefore gives the field a realistic, shared standard for comparing anonymization methods.

Key points

  • Corpus scale and labels: TAB holds 1,268 ECHR cases. Twelve law students annotated them. Each annotation records the entity category (PERSON, CODE, DATETIME, DEM, LOC, ORG, QUANTITY, MISC), the identifier type (DIRECT, QUASI, NO MASK), confidential attributes and co-reference links.
  • Privacy metrics: An entity only counts as protected if every one of its mentions is masked. Leaking a direct identifier is penalized more than leaking a quasi-identifier.
  • Baselines: Three approaches were tested: a generic neural NER model, a privacy-oriented NER-based de-identification system, and BERT sequence labelers fine-tuned on TAB.
  • Annotator disagreement: Among 4,299 unique masking disagreements, most were QUASI vs. NO MASK, typically for ORG, DATETIME, DEM and LOC. About 5% were DIRECT vs. QUASI, and none were DIRECT vs. NO MASK on identical spans.
  • Availability: The authors have released the corpus together with its annotation guidelines, the scripts used for evaluation and the baseline models at github.com/NorskRegnesentral/text-anonymization-benchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OsloOslo · Norway29 works · 3,700 citations🎓Universitat Rovira i VirgiliTarragona · Spain3 works · 182 citations🤝Norwegian Computing CenterOslo · Norway2 works · 364 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 685 works

Cite

Cite as Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez and Montserrat Batet. 2022. The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00458

@article{pilan2022text,
  title = {{The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization}},
  author = {Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez and Montserrat Batet},
  journal = {Computational Linguistics},
  year = {2022},
  doi = {10.1162/coli\_a\_00458},
  url = {https://doi.org/10.1162/coli_a_00458}
}

References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aberdeen, John, Samuel Bayer, Reyyan Yeniterzi, Ben Wellner, Cheryl Clark, David Hanauer, Bradley Malin, and Lynette Hirschman. 2010. The MITRE identification scrubber toolkit: Design, training, and assessment. International Journal of Medical Informatics, 79(12):849–859. https://doi.org/10 .1016/j.… find
  2. Anandan, Balamurugan and Chris Clifton. 2011. Significance of term relationships on anonymization. In Proceedings of the 2011 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Workshops, WI-IAT 2011, pages 253–256. https://doi.org/10 .1109/WI-IAT.2011… find
  3. Artstein, Ron and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. https://doi.org/10.1162/coli.07-034-R2 Barrett, Maria, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott, and Anders Søgaard. 2019. Adversarial r… ↗
  4. Bick, Eckhard and Anabela Barreiro. 2015. Automatic anonymisation of a new Portuguese-English parallel corpus in the legal-financial domains. Oslo Studies in Language, 7(1):101–124. https://doi.org/10.5617/osla.1460 Bier, Eric A., Richard Chow, Philippe Golle, Tracy H. King, and J. Staddon. 2009. T… — [doi:10.5617/osla.1460](https://doi.org/10.5617/osla.1460) ↗
  5. O’Reilly Media, Inc. Blodgett, Su Lin, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130. https://doi.org/10 .18…
  6. Bommasani, Rishi, Steven Wu, Zhiwei, and Alexandra K. Schofield. 2019. Towards private synthetic text generation. In NeurIPS 2019 Workshop on Machine Learning with Guarantees, 5 pages. find
  7. Chakaravarthy, Venkatesan T., Himanshu Gupta, Prasan Roy, and Mukesh K. Mohania. 2008. Efficient techniques for document sanitization. In Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM 2008, pages 843–852. find
  8. Chiu, Jason P. C. and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357–370. https://doi.org/10 .1162/tacl a 00104 Chow, Richard, Philippe Golle, and Jessica Staddon. 2008. Detecting privacy leaks using cor…
  9. Cumby, Chad M. and Rayid Ghani. 2011. A machine learning based system for semi-automatically redacting documents. In Proceedings of the Twenty-Third Conference on Innovative Applications of Artificial Intelligence, pages 1628–1635. https://doi.org/10.1609/aaai.v25i2.18851 Dernoncourt, Franck, Ji Yo… ↗
  10. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  11. Dwork, Cynthia and Aaron Roth. 2014. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3–4):211–407. https://doi.org/10 .1561/0400000042 Eder, Elisabeth, Ulrike Krieg-Holz, and Udo Hahn. 2020. CodE Alltag 2.0—A pseudonymized German-languag… find
  12. Elazar, Yanai and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11–21.
  13. Fernandes, Natasha, Mark Dras, and Annabelle McIver. 2019. Generalised differential privacy for text document processing. In Principles of Security and Trust - 8th International Conference, POST 2019, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2019, Proc… — [doi:10.1007/978-3-030-17138-4](https://doi.org/10.1007/978-3-030-17138-4) find
  14. GDPR. 2016. General Data Protection Regulation. European Union Regulation 2016/679. find
  15. Gearty, Conor A. 1993. The European Court of Human Rights and the protection of civil liberties: An overview. The Cambridge Law Journal, 52(1):89–127. https://doi.org/10.1017/S0008197300017256 Golle, Philippe. 2006. Revisiting the uniqueness of simple demographics in the US population. In Proceedin… — [doi:10.1017/s0008197300017256](https://doi.org/10.1017/s0008197300017256) ↗
  16. Hintze, Mike. 2017. Viewing the GDPR through a de-identification lens: A tool for compliance, clarification, and consistency. International Data Privacy Law, 8(1):86–101. https://doi.org/10.1093/idpl/ipx020 — [doi:10.1093/idpl/ipx020](https://doi.org/10.1093/idpl/ipx020) ↗
  17. HIPAA. 2004. The Health Insurance Portability and Accountability Act, U.S. Dept. of Labor, Employee Benefits Security Administration. find
  18. Honnibal, Matthew, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. find
  19. Huang, Yangsibo, Zhao Song, Danqi Chen, Kai Li, and Sanjeev Arora. 2020. TextHide: Tackling data privacy in language understanding tasks. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1368–1382. https://doi.org/10.18653/v1/2020 .findings-emnlp.123 Hundepool, Anco, J… ↗
  20. Johnson, Alistair E. W., Lucas Bulgarelli, and Tom J. Pollard. 2020. Deidentification of free-text medical records using pre-trained bidirectional transformers. In Proceedings of the ACM Conference on Health, Inference, and Learning, pages 214–221. https://doi.org/10.1145/3368555.3384455 Kasper, De… ↗
  21. Lee, Jinhyuk, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. https://doi.org/10.1093/bioinformatics/btz682, PubMed: 31501885 Lee, Ke… ↗
  22. Li, Ninghui, Tiancheng Li, and Suresh Venkatasubramanian. 2007. t-Closeness: Privacy beyond k-anonymity and l-diversity. In 23rd International Conference on Data Engineering (ICDE 2007), pages 106–115. https://doi.org/10.1109/ICDE.2007.367856 Li, Xuechen, Florian Tramèr, Percy Liang, and Tatsunori… — [doi:10.1109/icde.2007.367856](https://doi.org/10.1109/icde.2007.367856) ↗
  23. Li, Yitong, Timothy Baldwin, and Trevor Cohn. 2018. Towards robust and privacy-preserving text representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 25–30. https://doi.org/10.18653/v1/P18-2005 Lison, Pierre, Ildiko… ↗
  24. Liu, Zengjian, Buzhou Tang, Xiaolong Wang, and Qingcai Chen. 2017. De-identification of clinical notes via recurrent neural network and conditional random field. Journal of Biomedical Informatics, 75:S34–S42. https://doi.org/10.1016/j.jbi.2017.05.023, PubMed: 28579533 Loshchilov, Ilya and Frank Hut… ↗
  25. Marimon, Montserrat, Aitor Gonzalez-Agirre, Ander Intxaurrondo, Heidy Rodriguez, Jose Lopez Martin, Marta Villegas, and Martin Krallinger. 2019. Automatic de-identification of medical texts in Spanish: The MEDDOCAN track, corpus, guidelines, methods and evaluation of results. In IberLEF@ SEPLN, page… find
  26. McMahan, H. Brendan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2017. Learning differentially private recurrent language models. arXiv:1710.06963 [cs]. — [arXiv:1710.06963](https://arxiv.org/abs/1710.06963) find
  27. Medlock, Ben. 2006. An introduction to NLP-based textual anonymisation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), pages 1051–1056. Megyesi, Beáta, Lena Granstedt, Sofia Johansson, Julia Prentice, Dan Rosén, Carl-Johan Schenström, Gunlög… find
  28. Mendels, Omri. 2020. Custom NLP approaches to data anonymization. Towards Data Science. https://towardsdatascience.com/nlp -approaches-to-data-anonymization-1fb5bde6b929. Accessed: 2022-06-06. ↗
  29. Meystre, Stephane M., F. Jeffrey Friedlin, Brett R. South, Shuying Shen, and Matthew H. Samore. 2010. Automatic de-identification of textual documents in the electronic health record: A review of recent research. BMC Medical Research Methodology, 10(1):70. https://doi.org/10.1186/1471-2288-10-70, P… ↗
  30. Neamatullah, Ishna, Margaret M. Douglass, H. Lehman Li-wei, Andrew Reisner, Mauricio Villarroel, William J. Long, Peter Szolovits, George B. Moody, Roger G. Mark, and Gari D. Clifford. 2008. Automated de-identification of free-text medical records. BMC Medical Informatics and Decision Making, 8(1):3… find
  31. Rumbold, John Mark Michael and Barbara Pierscionek. 2017. The effect of the general data protection regulation on medical research. Journal of Medical Internet Research, 19(2):e47. https://doi.org/10 .2196/jmir.7108, PubMed: 28235748 Samarati, Pierangela. 2001. Protecting respondents’ identities in… — [doi:10.1109/69.971193](https://doi.org/10.1109/69.971193) find
  32. Samarati, Pierangela and Latanya Sweeney. 1998. Protecting privacy when disclosing information: k-Anonymity and its enforcement through generalization and suppression, Technical report, SRI International. Sánchez, David and Montserrat Batet. 2016. C-sanitized: A privacy model for document redaction… — [doi:10.1002/asi.23363](https://doi.org/10.1002/asi.23363) find
  33. Trieu, Lap Q., Trung-Nguyen Tran, Mai-Khiem Tran, and Minh-Triet Tran. 2017. Document sensitivity classification for data leakage prevention with Twitter-based document embedding and query expansion. In Proceedings of the 13th International Conference on Computational Intelligence and Security, page… — [doi:10.1109/cis.2017](https://doi.org/10.1109/cis.2017) find
  34. Mitchell, Palmer Martha S., Robert Belvin, Sameer S. Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A large training corpus for enhanced processing. In Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation, 8 pages. find
  35. Weitzenboeck, Emily, Pierre Lison, Malgorzata Agnieszka Cyndecka, and Malcolm Langford. 2022. The GDPR and unstructured data: Is anonymization possible? International Data Privacy Law, 12:184–206. https://doi.org/10 .1093/idpl/ipac008 Westin, Alan F. 1967. find
  36. Privacy and Freedom, Atheneum, New York. Xu, Qiongkai, Lizhen Qu, Chenchen Xu, and Ran Cui. 2019. Privacy-aware text rewriting. In Proceedings of the 12th International Conference on Natural Language Generation, pages 247–257. https://doi.org/10.18653/v1/W19-8633 Yang, Hui and Jonathan M. Garibaldi… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pilan-2022-text-anonymization-benchmark-tab-dedicated/