📄 Evaluating Encoder- and LLM-Based Approaches for Robust Indirect Personal Identifier Detection

Summary

Stripping explicit protected health information (PHI) from clinical notes does not remove all re-identification risk. Contextual details such as occupation, family situation, lifestyle, or named institutions can still single out a patient when linked with outside sources. The authors address the automatic detection of these Indirect Personal Identifiers (IPIs) at span level in English discharge summaries.

Using the IPI-annotated MIMIC-III corpus of Baroud et al. (2025), they compare three paradigms:

• fine-tuned encoder-only models;
• prompt-based frontier LLMs, including a two-stage variant that separates span finding from labelling;
• hybrid pipelines, where an LLM proposes candidate spans and a RoBERTa-large classifier assigns labels.

A fine-tuned RoBERTa-large performs best overall. It beats the earlier BERT-base baseline and far exceeds ChatGPT-5.2, and the hybrid setups also fall short of it. The authors argue that IPI detection is a distinct problem, marked by heavy class imbalance and very diverse expressions within each class. In this setting, larger models do not automatically give balanced per-class performance. Supervised encoders are therefore currently the most dependable basis for anonymization that goes beyond fixed PHI lists.

Key points

• Data: 100 de-identified MIMIC-III discharge summaries with 6199 annotated spans (inter-annotator agreement 0.87) across nine labels (BODY_DESC, SOCIO, DETAILS, DIRECT_ID, FAMILY, HEALTH_FCLT, RELATIVE_TIME, LFSTL, OTHER). The labels are dominated by RELATIVE_TIME and HEALTH_FCLT.
• Main result: RoBERTa-large reaches 0.906 micro-F1 and 0.724 macro-F1. ChatGPT-5.2 reaches 0.509 and 0.487, and the prior BERT-base baseline 0.78 and 0.50.
• Encoder choice: BioBERT, ClinicalBERT and ModernBERT brought no macro-level gains. QLoRA tuning of Qwen-14B did not consistently beat few-shot prompting of frontier LLMs.
• Hybrid bottleneck: The ChatGPT-5.2 filtering stage recovered only 0.671 micro and 0.743 macro span recall, for example 638 of 1003 RELATIVE_TIME spans. This caps what the hybrid pipeline can achieve.
• Evaluation setup: Relaxed span-level precision, recall and F1, with all documents split into chunks of up to 512 tokens.