Elamur Research⌕
☰
home / papers / 2026 / A framework for analyzing concept representations in neural models

A framework for analyzing concept representations in neural models

Summary

Interpretability work often assumes that human-interpretable concepts live in linear subspaces of a model's representation space. However, probing and concept-erasure studies define and evaluate such subspaces differently and rarely compare their estimation methods. The authors propose one framework that scores a candidate subspace along two axes using linear probing classifiers. Containment asks whether a concept can be decoded inside the subspace (retention) but not from its complement (leakage). Disentanglement asks whether another concept is absent from the subspace (purity) and survives when the subspace is removed (interference). They apply the framework to gender and profession in BERT and to phones and speakers in HuBERT, comparing five subspace estimators from different research communities.

Key findings:

  • Concept subspaces need not be unique, because redundant information can make retention and leakage both high.
  • The choice of estimator changes the conclusions.
  • LEACE performs best overall but leaks on unseen data.
  • Phone subspaces are well contained and disentangled. Speaker subspaces resist compact, generalizable containment.

The framework clarifies what claims about concept representations actually mean.

Key points

  • The four metrics are bounded by two references: full-space accuracy (best case) and a majority-class baseline (worst case). In erasure terms, leakage corresponds to guardedness and interference to collateral damage.
  • LEACE guarantees zero leakage on the data it was fit on, but this guarantee does not carry over to unseen speakers.
  • Speaker subspaces estimated from 40 speakers overfit. Using LibriSpeech train-clean-460 (1172 speakers) helps, but only modestly at dimensionality M = 40.
  • Raising M to 700 steadily improves speaker containment but lowers purity. Interference stays low up to M = 500, so phone information is robust to speaker erasure.
  • A layerwise LEACE analysis of HuBERT shows that early layers have higher speaker retention but also higher leakage. Code is available at github.com/burin-n/concept_space.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.conll-main.34

@inproceedings{naowarat2026framework,
  title = {{A framework for analyzing concept representations in neural models}},
  author = {Naowarat, Burin and Tang, Hao and Goldwater, Sharon},
  booktitle = {Proceedings of the 30th Conference on Computational Natural Language Learning},
  year = {2026},
  doi = {10.18653/v1/2026.conll-main.34},
  url = {https://doi.org/10.18653/v1/2026.conll-main.34}
}

References

41 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takanori Ashihara, Marc Delcroix, Takafumi Moriya, Kohei Matsuura, Taichi Asami, and Yusuke Ijima. 2024. What do self-supervised speech and speaker models learn? new findings from a cross model layer-wise analysis. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)… find
  2. Yonatan Belinkov. 2022. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics. find our card
  3. Yonatan Belinkov and James Glass. 2017. Analyzing hidden representations in end-to-end automatic speech recognition systems. In Advances in Neural Information Processing Systems, pages 2441–2451. find
  4. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. LEACE: Perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems (NeurIPS). find
  5. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems… find our card
  6. Marc-André Carbonneau, Julian Zaidi, Jonathan Boilard, and Ghyslain Gagnon. 2022. Measuring disentanglement: A review of metrics. IEEE Trans. Neural Netw. Learn. Syst. find
  7. Tyler A Chang, Zhuowen Tu, and Benjamin K Bergen. 2022. The geometry of multilingual language model representations. In Empirical Methods in Natural Language Processing (EMNLP). find
  8. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2021. WavLM: Large-scale self-supervised pre-training for full stack… find
  9. Grzegorz Chrupała, Bertrand Higy, and Afra Alishahi. 2020. Analyzing analytical methods: The case of phonology in neural models of spoken language. In Association for Computational Linguistics (ACL). find
  10. Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. 2019. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems (NeurIPS). find our card
  11. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Association for Computational Linguistics (ACL). find our card
  12. Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. 2024. Recurrent neural networks learn to store and generate sequences using non-linear representations. In BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. find
  13. Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. 2022. Discovering latent concepts learned in BERT. In International Conference on Representation Learning (ICLR). find
  14. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairnes… find
  15. Cian Eastwood, Andrei Liviu Nicolicioiu, Julius Von Kügelgen, Armin Kekić, Frederik Träuble, Andrea Dittadi, and Bernhard Schölkopf. 2022. DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability. In International Conference on Representation Learning (ICLR). find
  16. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. 2024. Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations. In Proceedings of the Third Conference on Causal Learning and Reasoning, pages 160–187. PMLR. find
  17. Michele Gubian, Ioana Krehan, Oli Liu, James Kirby, and Sharon Goldwater. 2025. Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech models. arXiv:2506.10855. — [arXiv:2506.10855](https://arxiv.org/abs/2506.10855) find
  18. Evan Hernandez and Jacob Andreas. 2021. The low-dimensional linear geometry of contextualized word representations. In CoNLL. find
  19. Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. 2018. Towards a definition of disentangled representations. arXiv:1812.02230. — [arXiv:1812.02230](https://arxiv.org/abs/1812.02230) find
  20. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460. find our card
  21. Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. 2024. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find
  22. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse Autoencoders Find Highly Interpretable Features in Language Models. In International Conference on Representation Learning (ICLR). find
  23. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Association for Computational Linguistics (ACL). find our card
  24. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In International Conference on Machine Learning (ICML). find
  25. Shauli Ravfogel, Yoav Goldberg, and Ryan Cotterell. 2023. Log-linear guardedness and its implications. In Association for Computational Linguistics (ACL). find
  26. Aravind Krishnan, Badr M Abdullah, and Dietrich Klakow. 2024. On the Encoding of Gender in Transformer-based ASR Representations. In Interspeech. find
  27. Oli Liu, Hao Tang, and Sharon Goldwater. 2023. Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces. In Interspeech. find
  28. Kinan Martin, Jon Gauthier, Canaan Breiss, and Roger Levy. 2023. Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration. In Proceedings of INTERSPEECH 2023. find
  29. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751. find our card
  30. Mukhtar Mohamed, Oli Danyi Liu, Hao Tang, and Sharon Goldwater. 2024. Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations. In Interspeech. find
  31. Hosein Mohebbi, Grzegorz Chrupała, Willem Zuidema, Afra Alishahi, and Ivan Titov. 2024. Disentangling Textual and Acoustic Features of Neural Speech Representations. arXiv:2410.03037. — [arXiv:2410.03037](https://arxiv.org/abs/2410.03037) find
  32. Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. 2022. Linear adversarial concept erasure. International Conference on Machine Learning (ICML). find
  33. Antoine Saillenfest and Pirmin Lemberger. 2025. Nonlinear Concept Erasure: a Density Matching Approach. In European Chapter of the Association for Computational Linguistics (EACL). find
  34. Shun Shao, Yftah Ziser, and Shay B. Cohen. 2023. Gold doesn’t always glitter: Spectral removal of linear and nonlinear guarded attribute information. In European Chapter of the Association for Computational Linguistics (EACL). find
  35. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Association for Computational Linguistics (ACL). find our card
  36. Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. 2020. A theory of usable information under computational constraints. In International Conference on Representation Learning (ICLR). find
  37. Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-Tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-Yi Lee. 2… find
  38. Vassil Panayotov, Guouo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). find
  39. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. 2025. The geometry of categorical and hierarchical concepts in large language models. In International Conference on Representation Learning (ICLR). find
  40. Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the geometry of large language models. In Causal Representation Learning Workshop at NeurIPS 2023. find our card
  41. Ankita Pasad, Bowen Shi, and Karen Livescu. 2023. Comparative layer-wise analysis of self-supervised speech models. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/naowarat-2026-framework-analyzing-concept-representations-neural/