Elamur Research⌕
☰
home / papers / 2026 / Birds of a Feather: Do Embedding Representations of Personal Information Flock Together?

Birds of a Feather: Do Embedding Representations of Personal Information Flock Together?

Paper · 2026 · Maria Irena Szawerna and Simon Dobnik

Summary

Automatic de-identification usually relies on embedding representations to find and classify personal information (PI). It is unclear, however, whether these representations reflect the categories that human annotators use. The authors investigate this by clustering embeddings of PI spans from Swedish L2 learner essays in SweLL-pilot and SweLL-gold, using the original, non-pseudonymized texts. They compare the resulting clusters with expert labels at both levels of a hierarchical PI taxonomy.

Span embeddings come from three models: a static FastText model, KB-BERT, and the word-in-context model XL-LEXEME. The embeddings are reduced with UMAP and clustered with four algorithms. The best setup per model is chosen by silhouette score, then evaluated with homogeneity, completeness, v-score and normalized per-cluster entropy.

Contextual embeddings form clearly better clusters than FastText. XL-LEXEME leaves the fewest outliers and produces very pure clusters, but it mostly groups spans by the coarse general classes rather than the fine-grained ones. KB-BERT follows the detailed labels more closely. Overall, the results show only partial agreement between model representations and the human PI taxonomy. This is relevant both for choosing models and for revising label sets in de-identification work.

Key points

  • Data: 947 essays (301,095 tokens) contain 3,076 annotated PI spans (3,348 tokens).
    • The taxonomy has 7 general and 37 detailed categories; 32 detailed categories occur in the data.
    • The class distribution is strongly skewed.
  • Models and span embeddings:
    • Kubord-fasttext (Dagens Nyheter 2010–2024), 300 dimensions, as a non-contextual baseline.
    • KB/bert-base-swedish-cased, 768 dimensions, using last-layer embeddings averaged over multi-token spans.
    • pierluigic/xl-lexeme, based on XLM-RoBERTa-large, 1024 dimensions, which directly embeds whole spans.
  • Clustering: HDBSCAN, Affinity Propagation, Mean Shift and Agglomerative Clustering (scikit-learn) were tested after UMAP reduction.
    • HDBSCAN won for every embedding type.
    • Silhouette scores were 0.83 for FastText, 0.68 for KB-BERT and 0.72 for XL-LEXEME.
  • Homogeneity: For the contextual embeddings, homogeneity rises noticeably when clusters are compared with general labels instead of detailed ones; this does not happen for FastText.
    • For XL-LEXEME, the share of near-zero-entropy clusters jumps sharply at the general level.
  • Outliers: XL-LEXEME leaves far fewer spans as outliers than the other models.
    • For personal_name, outliers are about 14% with XL-LEXEME, 35% with KB-BERT and 58% with FastText.
    • KB-BERT's cluster count comes closest to the number of detailed labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GothenburgGothenburg · Sweden34 works · 1,625 citations

Also on the byline: Språkbanken Text.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Maria Irena Szawerna and Simon Dobnik. 2026. Birds of a Feather: Do Embedding Representations of Personal Information Flock Together?. Proceedings of the Joint Workshop on Legal and Ethical Issues in Human Language Technologies and Computational Approaches to Language Data Pseudonymization, Anonymization, De-identification, and Data Privacy (LEGAL2026 and CALD-pseudo 2026) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4ohwx42xhvzy

@inproceedings{szawerna2026birds,
  title = {{Birds of a Feather: Do Embedding Representations of Personal Information Flock Together?}},
  author = {Maria Irena Szawerna and Simon Dobnik},
  booktitle = {Proceedings of the Joint Workshop on Legal and Ethical Issues in Human Language Technologies and Computational Approaches to Language Data Pseudonymization, Anonymization, De-identification, and Data Privacy (LEGAL2026 and CALD-pseudo 2026) @ LREC 2026},
  year = {2026},
  doi = {10.63317/4ohwx42xhvzy},
  url = {https://doi.org/10.63317/4ohwx42xhvzy}
}

References

12 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pierluigi Cassotti, Lucia Siciliani, Marco DeGemmis, Giovanni Semeraro, and Pierpaolo Basile. 2023. XL-LEXEME: WiC pretrained model for cross-lingual LEXical sEMantic changE. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 15… find
  2. Luc Steels and Tony Belpaeme. 2005. Coordinating perceptually grounded categories through language: a case study for colour. Behavioral and Brain Sciences, 28(4):469–489. find
  3. Maria Irena Szawerna, Simon Dobnik, Ricardo Muñoz Sánchez, Therese Lindström Tiedemann, and Elena Volodina. 2024. Detecting Personal Identifiable Information in Swedish Learner Essays. In Proceedings of the Workshop on Computational Approaches to Language Data Pseudonymization (CALD-pseudo 2024), pa… find
  4. Martin Malmsten, Love Börjeson, and Chris Haffenden. 2020. Playing with Words at the National Library of Sweden – Making a Swedish BERT.
  5. Språkbanken Text. 2025a. Kubord-fasttext - dagens nyheter 2010–2024 - token. find
  6. Maria Irena Szawerna, Simon Dobnik, Ricardo Muñoz Sánchez, and Elena Volodina. 2025. The Devil’s in the Details: the Detailedness of Classes Influences Personal Information Detection and Labeling. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Confere… find
  7. Språkbanken Text. 2025b. Swell. find
  8. Elena Volodina. 2024. On two swell learner corpora – swell-pilot and swell-gold. In Proceedings of the Huminfra Conference (HiC 2024), 10-11 January, 2024, Gothenburg, Sweden, pages 83–94, Linköping. Linköping University Press. find
  9. Thomas Vakili, Anastasios Lamproudis, Aron Henriksson, and Hercules Dalianis. 2022. Downstream Task Performance of BERT Models Pre-Trained Using Automatically De-Identified Clinical Data. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4245–4252, Marseille, Franc… find
  10. Elena Volodina, Yousuf Ali Mohammed, Sandra Derbring, Arild Matsson, and Beata Megyesi. 2020. Towards Privacy by Design in Learner Corpora Research: A Case of On-the-fly Pseudonymization of Swedish Learner Essays. In Proceedings of the 28th International Conference on Computational Linguistics, page… find
  11. Elena Volodina, Simon Dobnik, Therese Lindström Tiedemann, Ricardo Muñoz Sánchez, Maria Irena Szawerna, and Lisa Södergård. 2025. Towards Shared Standards for Pseudonymization of Research Data. In Proceedings of the 2nd Huminfra Conference. find
  12. Michael L. Waskom. 2021. seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/szawerna-2026-birds-feather-do-embedding-representations/