Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.tacl-1.10.pdf

Summary

Probing is a common way to inspect neural language models: a small classifier is trained to recover a linguistic property from hidden representations. However, a successful probe only shows that the information can be extracted. It says nothing about whether the model actually relies on it. Elazar et al. propose Amnesic Probing, a counterfactual test of usage. The target property is erased from the representation, and the authors measure how much the model's behavior on its main task degrades. The erasure uses Iterative Nullspace Projection (INLP), which removes linearly decodable information without retraining the encoder. Applying this to BERT-base for word prediction, they study properties such as part-of-speech and dependency labels. Their central finding is that high probing accuracy does not predict how important a property is for the task. Usage also differs between masked and non-masked inputs, and individual labels within a property matter to different degrees. The work is a warning against drawing causal or behavioral conclusions from probing scores alone. It also offers a reusable intervention-based analysis tool.

Key points

• How INLP erases a property: It repeatedly trains linear classifiers (linear SVM) to predict the property and projects the representations onto each classifier's nullspace. This continues until a probe scores within about one point of majority accuracy on the development set.
• Two controls:
  • Rand removes the same number of directions, chosen at random, to account for generic information loss.
  • Selectivity reattaches gold property values as learned 32-dimensional vectors and fine-tunes the following layers to check whether performance can be restored.
• Masked vs. non-masked: Properties like POS and dependency labels are used differently when the token is masked (the LM training regime) than when it is visible (the regime typical for probing and fine-tuning).
• NLP pipeline claim revisited: The authors re-examine the claim of Tenney et al. (2019a) that BERT's layers mirror a classical NLP pipeline. Their layer-wise amnesic analysis gives a new interpretation, but it covers BERT-base, whereas Tenney et al. studied BERT-Large.
• Scope and availability: Published in TACL vol. 9 (2021), pp. 160–175. The method removes only linear information, and the authors call for non-linear removal methods. Code is at github.com/yanaiela/amnesic_probing.