Elamur Research⌕
☰
home / papers / 2026 / CAPID: Context-Aware PII Detection for Question-Answering Systems

CAPID: Context-Aware PII Detection for Question-Answering Systems

Paper · 2026 · Mariia Ponomarenko, Sepideh Abedini, Masoumeh Shafieinejad et al.

Summary

Question-answering assistants often receive queries in which users mention personal details. Standard PII tools mask all of them, even when some details (such as a job or a health condition) are needed for a good answer. Sending raw queries to third-party LLMs is a privacy risk. CAPID addresses this by running a small, locally hosted language model that finds PII spans, assigns each one a type, and decides whether it matters for the question. Only the irrelevant spans are masked before the query goes to an external LLM.

No public dataset labels this kind of question-dependent relevance, so the authors built a synthetic one. GPT-4.1-mini and GPT-5 generated topics from pairs of PII types, then wrote contexts that mix relevant and peripheral personal details, followed by matching questions. Llama models fine-tuned on this data classified relevance better than GPT-4.1-mini on annotated real Reddit queries. Selective masking also kept answers more useful than full redaction. This shows that privacy protection and answer quality can be balanced.

Key points

  • PII taxonomy: 15 fine-grained types, including occupation, health, location, belief, code and appearance. Pairing 13 of these types gave 78 combinations, which expanded into 15,600 topic–subtopic pairs. After de-duplication, 11,663 unique (PIIType1, PIIType2, subtopic) triplets remained.
  • Models: Llama-3.1-8B and Llama-3.2-3B were fine-tuned. The 8B LoRA model is released on Hugging Face as ponoma16/capid-llama8b-lora, and the code and dataset are on GitHub.
  • Relevance accuracy on Reddit: the fine-tuned Llama-3.1-8B scored 0.7994, compared with 0.6896 for GPT-4.1-mini. GPT-4.1-mini over-predicted spans, giving high recall but lower precision.
  • Downstream utility: answers were generated by GPT-4 and judged by both GPT-4 and Claude Sonnet 4.5. Masking only low-relevance PII was preferred over full masking in 0.73–0.80 of cases. The pretrained-SLM baseline of Ngong et al. (2025) reached only 0.43–0.58.
  • Limitations: performance drops on long contexts and on questions without cues about what information is needed. Relevance labels are binary rather than continuous sensitivity scores, and the method does not apply expert, domain-specific rules (e.g., legal or medical).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WaterlooWaterloo · Canada80 works · 15,030 citations🔬Vector InstituteToronto · Canada37 works · 5,703 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 487 works

Cite

Cite as Mariia Ponomarenko, Sepideh Abedini, Masoumeh Shafieinejad, D. B. Emerson, Shubhankar Mohapatra and Xi He. 2026. CAPID: Context-Aware PII Detection for Question-Answering Systems. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-srw.23

@inproceedings{ponomarenko2026capid,
  title = {{CAPID: Context-Aware PII Detection for Question-Answering Systems}},
  author = {Mariia Ponomarenko, Sepideh Abedini, Masoumeh Shafieinejad, D. B. Emerson, Shubhankar Mohapatra and Xi He},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.23},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.23}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI4Privacy. 2022. Pii masking 300k dataset. https://huggingface.co/datasets/ai4privacy/pii-masking-300k. Accessed: 2025-10-03. ↗
  2. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Rom… — SantaCoder: don't reach for the stars!
  3. Michael Han Daniel Han and Unsloth team. 2023. Unsloth. find
  4. Yao Dou, Isadora Krsek, Tarek Naous, Anubha Kabra, Sauvik Das, Alan Ritter, and Wei Xu. 2024. Reducing privacy risks in online self-disclosures with language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13732– 13754… find
  5. Rajitha Hathurusinghe, Isar Nejadgholi, and Miodrag Bolic. 2021. A privacy-preserving approach to extraction of personal information through automatic annotation and federated learning. In Proceedings of the Third Workshop on Privacy in Natural Language Processing, pages 36–45, Online. Association f… find
  6. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  7. Shalini Jangra, Suparna De, Nishanth Sastry, and Saeed Fadaei. 2025. Protecting vulnerable voices: Synthetic dataset generation for self-disclosure detection. Preprint, arXiv:2507.22930. — [arXiv:2507.22930](https://arxiv.org/abs/2507.22930) find
  8. Iyadh Ben Cheikh Larbi, Aljoscha Burchardt, and Roland Roller. 2022. Which anonymization technique is best for which nlp task? – it depends. a systematic study on clinical text processing. Preprint, arXiv:2209.00262. — [arXiv:2209.00262](https://arxiv.org/abs/2209.00262) find
  9. Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On LLMs-driven synthetic data generation, curation, and evaluation: A survey. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11065–11082, Bangkok, Thailand. Association for Comput…
  10. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Beguelin. 2023. Analyzing Leakage of Personally Identifiable Information in Language Models . In 2023 IEEE Symposium on Security and Privacy (SP), pages 346– 363, Los Alamitos, CA, USA. IEEE Computer Society.
  11. Amazon. 2025. Amazon comprehend. https://aws.amazon.com/comprehend/. Accessed: 2025-10-03. ↗
  12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  13. Microsoft. 2021. Presidio. https://microsoft.github.io/ presidio/. Accessed: 2025-10-03. ↗
  14. Ivoline C. Ngong, Swanand Ravindra Kadhe, Hao Wang, Keerthiram Murugesan, Justin D. Weisz, Amit Dhurandhar, and Karthikeyan Natesan Ramamurthy. 2025. Protecting users from themselves: Safeguarding contextual privacy in interactions with conversational agents. In Findings of the Association for Compu… find
  15. Helen Nissenbaum. 2004. Privacy as contextual integrity. Washington Law Review, 79. find
  16. Liu, Quanquan Gu, Haifeng Chen, Wei Wang, and Wei Cheng. 2024. Large language models can be contextual privacy protection learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14179–14201, Miami, Florida, USA. Association for Computational Linguis… find
  17. Anwesan Pal, Radhika Bhargava, Kyle Hinsz, Jacques Esterhuizen, and Sudipta Bhattacharya. 2024. The empirical impact of data sanitization on language models. Preprint, arXiv:2411.05978. — [arXiv:2411.05978](https://arxiv.org/abs/2411.05978) find
  18. Anthi Papadopoulou, Pierre Lison, Lilja Øvrelid, and Ildikó Pilán. 2022a. Bootstrapping text anonymization models with distant supervision. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4477–4487, Marseille, France. European Language Resources Association. find
  19. Anthi Papadopoulou, Yunhao Yu, Pierre Lison, and Lilja Øvrelid. 2022b. Neural text sanitization with explicit measures of privacy risk. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Nat… find
  20. Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet. 2022. The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics, 48(4):1053–1101.
  21. Kambiz Saffarizadeh, Maheshwar Boodraj, and Tawfiq Alashoor. 2018. Conversational assistants: Investigating privacy concerns, trust, and self-disclosure. In ICIS 2017 Proceedings, Proceedings of the International Conference on Information Systems. Association for Information Systems. AIS Electronic… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ponomarenko-2026-capid-context-aware-pii-detection/