Elamur Research
☰
home / papers / 2022 / Probing Classifiers: Promises, Shortcomings, and Advances

Probing Classifiers: Promises, Shortcomings, and Advances

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.cl-1.7.pdf

Summary

This squib, a short critical review, examines probing classifiers, a widely used way of interpreting neural NLP models. A probe is an auxiliary classifier that predicts a linguistic property, such as part-of-speech tags, from a model's internal representations. Success is usually read as evidence that the model encodes that property. The author introduces no new data or experiments. Instead, he formalizes the framework, separating the original model, task and dataset from the probing classifier, task and dataset, and frames probing as estimating mutual information between representations and the property.

He then surveys known weaknesses and proposed remedies: missing baselines and controls, the choice and complexity of the probe, the purely correlational nature of probing results, confounding effects where datasets and tasks become entangled, and the requirement that target properties be specified beforehand. The main conclusion is that absolute claims about representation quality are hard to justify. Relative claims, such as how easily a property can be extracted, are defensible with proper controls and metrics. Claims about what the model actually uses require causal interventions. The paper serves as a practical checklist for designing probing studies.

Key points

  • The review was published in Computational Linguistics (DOI 10.1162/COLI_a_00422) under CC BY-NC-ND 4.0. Figure 1 unifies the basic and additional components of probing into one notation.
  • Baselines and controls discussed:
    • majority baselines, static word embeddings and randomly initialized models (random features already yield surprisingly strong probe accuracy);
    • skylines, such as human performance or state-of-the-art results;
    • control tasks and selectivity (Hewitt and Liang 2019);
    • control functions and information gain (Pimentel et al. 2020b);
    • control datasets (Ravichander, Belinkov, and Hovy 2021);
    • MDL probing (Voita and Titov 2020).
  • Recommendations:
    • Define all components explicitly and set lower and upper bounds.
    • Use control tasks for word-level properties and control datasets for sentence-level ones.
    • Measure probe complexity or the accuracy–complexity trade-off.
    • Use interventions to test whether the model actually relies on the information.
    • Consider parameter-free probes where possible.
  • Limitation of pre-defined properties: probing depends on existing annotated datasets, mostly English. Michael, Botha, and Tenney (2020) partly address this by learning latent clusters that reveal unannotated categories.
  • Few cases where probing improved models: decoupled QA layers (Cao et al. 2020), efficient transfer learning (Dalvi et al. 2020) and phonetic multitask supervision in ASR (Krishna, Toshniwal, and Livescu 2019).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

263 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 256 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00422

@article{belinkov2022probing,
  title = {{Probing Classifiers: Promises, Shortcomings, and Advances}},
  author = {Belinkov, Yonatan},
  journal = {Computational Linguistics},
  year = {2022},
  doi = {10.1162/coli\_a\_00422},
  url = {https://doi.org/10.1162/coli_a_00422}
}

References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adi, Yossi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. CoRR, abs/1608.04207. find our card
  2. Adi, Yossi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations (ICLR), Toulon. find our card
  3. Alain, Guillaume and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644v3. find our card
  4. Bau, Anthony, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations, New Orleans, LA. find
  5. Belinkov, Yonatan. 2018. On Internal Language Representations in Deep Learning: An Analysis of Machine Translation and Speech Recognition. Ph.D. thesis, Massachusetts Institute of Technology. find
  6. Belinkov, Yonatan, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017a. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872, Vancouver. https:… find our card
  7. Belinkov, Yonatan and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72. https://doi.org/10.1162 /tacl a 00254 Belinkov, Yonatan, Lluı́s Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Gl… find our card
  8. Cao, Qingqing, Harsh Trivedi, Aruna Balasubramanian, and Niranjan Balasubramanian. 2020. DeFormer: Decomposing pre-trained transformers for faster question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4487–4497, Online. https://doi.org/… find
  9. Chrupała, Grzegorz and Afra Alishahi. 2019. Correlating neural and symbolic representations of language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2952–2962, Florence. https://doi .org/10.18653/v1/P19-1283 Chrupała, Grzegorz, Bertrand Higy, and… find
  10. Clark, Kevin, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? An analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence. https://doi .org/10.18653/v1/W19-4… find
  11. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  12. Edmonds, Jack. 1967. Optimum branchings. Journal of Research of the National Bureau of Standards B, 71(4):233–240. find
  13. Elazar, Yanai, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175. https://doi.org/10.1162 /tacl a 00359 Ettinger, Allyson, Ahmed Elgohary, and Philip Re… find
  14. Feder, Amir, Nadav Oved, Uri Shalit, and Roi Reichart. 2021. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333–386. https://doi .org/10.1162/coli a 00404 Giulianelli, Mario, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema… find
  15. Hall Maudslay, Rowan, Josef Valvoda, Tiago Pimentel, Adina Williams, and Ryan Cotterell. 2020. A tale of a probe and a parser. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7389–7395, Online. https://doi.org/10.18653/v1/2020 .acl-main.659 Hewitt, J… ↗
  16. Htut, Phu Mon, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do attention heads in BERT track syntactic dependencies? arXiv preprint arXiv:1911.12246. — [arXiv:1911.12246](https://arxiv.org/abs/1911.12246) find
  17. Hupkes, Dieuwke, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and ’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907–926. https://doi.org/10.1613/jair.1.11196 Köhn, Arne. 2015. Wha… ↗
  18. Kriegeskorte, Nikolaus, Marieke Mur, and Peter Bandettini. 2008. Representational similarity analysis—connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:4. https://doi .org/10.3389/neuro.06.004.2008 Krishna, Kalpesh, Shubham Toshniwal, and Karen Livescu. 2019. Hier… — [doi:10.3389/neuro.06.004.2008](https://doi.org/10.3389/neuro.06.004.2008) find
  19. Lakretz, Yair, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… find
  20. Lepori, Michael and R. Thomas McCoy. 2020. Picking BERT’s brain: Probing for linguistic dependencies in contextualized embeddings using representational similarity analysis. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3637–3651, International Committee on… find
  21. Lovering, Charles, Rohan Jha, Tal Linzen, and Ellie Pavlick. 2021. Predicting inductive biases of pre-trained models. In International Conference on Learning Representations. Mareček, David and Rudolf Rosa. 2019. From balustrades to Pierre Vinken: Looking for syntax in transformer self-attentions.… find
  22. Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6792–6812, Online. https://doi .org/10.18653/v1/2020.emnlp-main.552 Pimentel, Tiago, Naomi Saphra, Adina Williams, and Ryan Cotterell. 2020a. Pareto probing: Trading off accuracy for complexity. In Proceedings of the 2020… find
  23. Rogers, Anna, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. https://doi.org/10.1162/tacl\_a\_00349 ↗ our card
  24. Shi, Xing, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–1534, Austin, TX. https://doi.org/10 .18653/v1/D16-1159 Tamkin, Alex, Trisha Singh, Davide Giovanardi… find our card
  25. Tenney, Ian, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In International Conference on Le… find our card
  26. Tucker, Mycal, Peng Qian, and Roger Levy. 2021. What if this modified that? Syntactic interventions with counterfactual embeddings. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 862–875, Online. https://doi .org/10.18653/v1/2021.findings -acl.76 Vanmassenhove,… find
  27. Veldhoen, Sara, Dieuwke Hupkes, and Willem H. Zuidema. 2016. Diagnostic classifiers revealing how neural networks process hierarchical structure. In CoCo@ NIPS, Barcelona. find
  28. Vig, Jesse, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388–12401, Curran Associates, Inc… find
  29. Voita, Elena and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183–196, Online. https://doi.org/10.18653/v1 /2020.emnlp-main.14 Wallace, Eric, Matt Gardner, and… find our card
  30. Zhang, Kelly and Samuel Bowman. 2018. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 359–361, Brussels. https://doi.o… ↗