Elamur Research
☰
home / papers / 2016 / Understanding intermediate layers using linear classifier probes

Understanding intermediate layers using linear classifier probes

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1610.01644

Summary

Deep neural networks are often called "black boxes", and the authors propose a simple way to look inside. A separate linear classifier, called a "probe", is trained on each intermediate layer. It tries to predict the correct class using only the features of that layer. Gradients from the probes are not propagated back into the model itself, so the measurements do not affect its training. The authors compare the probes to thermometers placed at many points at once. The method was tested on convolutional networks for MNIST and ImageNet, including Inception v3 and ResNet-50. The main observation is that the linear separability of features increases with depth almost strictly monotonically. The loss function explicitly requires this only of the last layer, so on intermediate layers it is a side effect. This does not contradict the data processing inequality: deeper layers do not contain more information, but they present it in a form that is easier to compute with. Probes also help to detect training problems that are not visible from the overall loss function alone. This work became one of the foundations of the widespread probing approach to analyzing representations, including in NLP.

Key points

  • A probe is softmax(W·h + b) over the features of layer k. It is trained with cross-entropy separately from the model, and backpropagation from it into the network is blocked. Because the probe is linear, training it is a convex problem.
  • For Inception v3, after roughly two weeks of training, the probe error decreases smoothly and almost monotonically from the lower layers to the upper ones. Early in training, the auxiliary head was slightly ahead of the main one.
  • A diagnostic example: a deliberately overly deep network of 128 fully connected layers with 128 neurons each, trained on MNIST. It has a long skip connection that bypasses the first half of the layers. The probes showed that this half remains "dead": the weights are nonzero, but no useful signal passes through it.
  • The theoretical justification relies on the Data Processing Inequality. The idea is illustrated with the example of a photo of a lion: a hex dump contains more information than the image, but it is of little use to a human.
  • Among future directions, the authors name ResNet-101, VGG-16/19, RNNs, GANs and adversarial examples. They consider multilayer probes premature for now, because such probes lose convexity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

239 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 228 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1610.01644 · arXiv 1610.01644

@misc{alain2016understanding,
  title = {{Understanding intermediate layers using linear classifier probes}},
  author = {Alain, Guillaume and Bengio, Yoshua},
  year = {2016},
  doi = {10.48550/arxiv.1610.01644},
  eprint = {1610.01644},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1610.01644}
}

References

28 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alain, G. and Bengio, Y. (2016). Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644. — [arXiv:1610.01644](https://arxiv.org/abs/1610.01644) find
  2. Arras, L., Montavon, G., Müller, K.-R., and Samek, W. (2017). Explaining recurrent neural network predictions in sentiment analysis. arXiv preprint arXiv:1706.07206. find
  3. Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140. find
  4. Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., and Roli, F. (2013). Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 387–402. Springer. find
  5. Binder, A., Montavon, G., Lapuschkin, S., Müller, K.-R., and Samek, W. (2016). Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pages 63–71. Springer. find
  6. Chollet, F. et al. (2015). Keras. https://github.com/fchollet/keras. ↗
  7. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. (2014). Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pages 647–655. find
  8. Dosovitskiy, A. and Brox, T. (2016). Inverting visual representations with convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4829–4837. find
  9. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680. find
  10. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770– 778. find
  11. Jarrett, K., Kavukcuoglu, K., Lecun, Y., et al. (2009). What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th International Conference on Computer Vision, pages 2146–2153. IEEE. find
  12. Jastrzebski, S., Arpit, D., Ballas, N., Verma, V., Che, T., and Bengio, Y. (2017). Residual connections encourage iterative inference. arXiv preprint arXiv:1710.04773. — [arXiv:1710.04773](https://arxiv.org/abs/1710.04773) find
  13. Lapuschkin, S., Binder, A., Montavon, G., Müller, K.-R., and Samek, W. (2016). Analyzing classifiers: Fisher vectors and deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2912–2920. find
  14. Larsson, G., Maire, M., and Shakhnarovich, G. (2016). Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648. — [arXiv:1605.07648](https://arxiv.org/abs/1605.07648) find
  15. Mahendran, A. and Vedaldi, A. (2015). Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5188–5196. find
  16. Mahendran, A. and Vedaldi, A. (2016). Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3), 233–255. find
  17. Montavon, G., Braun, M. L., and Müller, K.-R. (2011). Kernel analysis of deep networks. Journal of Machine Learning Research, 12(Sep), 2563–2581. find
  18. Raghu, M., Yosinski, J., and Sohl-Dickstein, J. (2017a). Bottom up or top down? dynamics of deep representations via canonical correlation analysis. arxiv. find
  19. Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. (2017b). Svcca: Singular vector canonical correlation analysis for deep understanding and improvement. arXiv preprint arXiv:1706.05806. — [arXiv:1706.05806](https://arxiv.org/abs/1706.05806) find
  20. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3), 211–252. find our card
  21. Singh, S., Hoiem, D., and Forsyth, D. (2016). Swapout: Learning an ensemble of deep architectures. In Advances In Neural Information Processing Systems, pages 28–36. find
  22. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199. find our card
  23. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–9. find
  24. Veit, A., Wilber, M. J., and Belongie, S. (2016). Residual networks behave like ensembles of relatively shallow networks. In Advances in Neural Information Processing Systems, pages 550– 558. find
  25. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048–2057. find our card
  26. Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014). How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320–3328. find
  27. Zeiler, M. D. and Fergus, R. (2014). Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer. find our card
  28. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530. find