Elamur Research
☰
home / papers / 2017 / Understanding Black-box Predictions via Influence Functions

Understanding Black-box Predictions via Influence Functions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1703.04730

Summary

Explanation methods for black-box models usually ask how a fixed, already-trained model reaches a particular output. Koh and Liang ask a different question: which training examples caused that output? Retraining the model once for every removed or changed example would answer it but is far too expensive. Instead, they borrow influence functions from robust statistics. An influence function estimates in closed form how the learned parameters, and the loss on a given test example, would change if one training point were slightly up-weighted, removed or perturbed. To make this practical for models with many parameters, the authors never build or invert the Hessian directly. They estimate the needed inverse-Hessian product with implicit Hessian-vector products, using conjugate gradients or a stochastic approximation, so the model only needs to supply gradients and Hessian-vector products. Tests on logistic regression, SVMs and convolutional networks show the estimates closely match actual retraining. They stay useful even when the model is non-convex or non-differentiable. This gives a single training-data-centred tool for understanding, debugging and attacking models.

Key points

  • The core quantity is I_up,loss = −∇L(z_test)ᵀ H⁻¹ ∇L(z). It differs from Euclidean nearest-neighbour similarity in two ways: training points with high loss get more weight, and the Hessian accounts for how much the other training points resist the change. In a logistic regression that separates 1s from 7s on MNIST, nearest neighbours miss harmful training images that share the test image's label.
  • The same framework covers small changes to input features, which shows which features of a training point drive a given test prediction. It also covers label changes and discrete data such as NLP inputs.
  • The approach scales because you compute one vector, s_test, per test point and then score every training point with a single dot product.
  • Influence was used to build training-set attacks that are visually indistinguishable from the original images and flip test predictions of an Inception-based dog-vs-fish classifier. It was also used to find mislabelled training examples. One limitation the authors raise: influence functions only describe small, local changes, so the effect of removing a large group of data (such as a whole subpopulation) is still an open question.
  • Code and data for reproducing the experiments are on GitHub (bit.ly/gt-influence) and Codalab (bit.ly/cl-influence). The paper appeared at ICML 2017.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

221 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 216 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1703.04730 · arXiv 1703.04730

@misc{koh2017understanding,
  title = {{Understanding Black-box Predictions via Influence Functions}},
  author = {Koh, Pang Wei and Liang, Percy},
  year = {2017},
  doi = {10.48550/arxiv.1703.04730},
  eprint = {1703.04730},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1703.04730}
}

References

52 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I. J., Harp, A., Irving, G., Isard, M., Jia, Y., Józefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D. G., Olah… find our card
  2. Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., and Venkatasubramanian, S. Auditing black-box models for indirect influence. arXiv preprint arXiv:1602.07043, 2016. — [arXiv:1602.07043](https://arxiv.org/abs/1602.07043) find
  3. Agarwal, N., Bullins, B., and Hazan, E. Second order stochastic optimization in linear time. arXiv preprint arXiv:1602.03943, 2016. — [arXiv:1602.03943](https://arxiv.org/abs/1602.03943) find
  4. Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., and Suh, J. Modeltracker: Redesigning performance analysis tools for machine learning. In Conference on Human Factors in Computing Systems (CHI), pp. 337–346, 2015. find
  5. Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Machine Learning, 79(1):151– 175, 2010. find
  6. Biggio, B., Nelson, B., and Laskov, P. Support vector machines under adversarial label noise. ACML, 20:97–112, 2011. find
  7. Biggio, B., Nelson, B., and Laskov, P. Poisoning attacks against support vector machines. In International Conference on Machine Learning (ICML), pp. 1467–1474, 2012. find
  8. Cadamuro, G., Gilad-Bachrach, R., and Zhu, X. Debugging machine learning models. In ICML Workshop on Reliable Machine Learning in the Wild, 2016. find
  9. Chatterjee, S. and Hadi, A. S. Influential observations, high leverage points, and outliers in linear regression. Statistical Science, pp. 379–393, 1986. find
  10. Christmann, A. and Steinwart, I. On robustness properties of convex risk minimization methods for pattern recognition. Journal of Machine Learning Research (JMLR), 5(0):1007–1034, 2004. find
  11. Cook, R. D. Detection of influential observation in linear regression. Technometrics, 19:15–18, 1977. find
  12. Cook, R. D. Assessment of local influence. Journal of the Royal Statistical Society. Series B (Methodological), pp. 133–169, 1986. find
  13. Cook, R. D. and Weisberg, S. Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22:495–508, 1980. find
  14. Cook, R. D. and Weisberg, S. Residuals and influence in regression. New York: Chapman and Hall, 1982. find
  15. Datta, A., Sen, S., and Zick, Y. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In Security and Privacy (SP), 2016 IEEE Symposium on, pp. 598–617, 2016. find
  16. Debruyne, M., Hubert, M., and Suykens, J. A. Model selection in kernel based regression using the influence function. Journal of Machine Learning Research (JMLR), 9 (0):2377–2400, 2008. find
  17. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. Decaf: A deep convolutional activation feature for generic visual recognition. In International Conference on Machine Learning (ICML), volume 32, pp. 647–655, 2014. find
  18. Frénay, B. and Verleysen, M. Classification in the presence of label noise: a survey. IEEE Transactions on Neural Networks and Learning Systems, 25:845–869, 2014. find
  19. Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015. find our card
  20. Goodman, B. and Flaxman, S. European union regulations on algorithmic decision-making and a “right to explanation”. arXiv preprint arXiv:1606.08813, 2016. — [arXiv:1606.08813](https://arxiv.org/abs/1606.08813) find
  21. Hampel, F. R. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(346):383–393, 1974. find
  22. Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., and Stahel, W. A. Robust Statistics: The Approach Based on Influence Functions. Wiley, 1986. find
  23. Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I., and Tygar, J. Adversarial machine learning. In Proceedings of the 4th ACM workshop on Security and artificial intelligence, pp. 43–58, 2011. find
  24. Jaeckel, L. A. The infinitesimal jackknife. Unpublished memorandum, Bell Telephone Laboratories, Murray Hill, NJ, 1972. find
  25. Kabra, M., Robie, A., and Branson, K. Understanding classifier errors by examining influential neighbors. In Computer Vision and Pattern Recognition (CVPR), pp. 3917– 3925, 2015. find
  26. Kansagara, D., Englander, H., Salanitro, A., Kagen, D., Theobald, C., Freeman, M., and Kripalani, S. Risk prediction models for hospital readmission: a systematic review. JAMA, 306(15):1688–1698, 2011. find
  27. Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. find
  28. Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1097–1105, 2012. find
  29. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. find
  30. Li, B., Wang, Y., Singh, A., and Vorobeychik, Y. Data poisoning attacks on factorization-based collaborative filtering. In Advances in Neural Information Processing Systems (NeurIPS), 2016a. find
  31. Li, J., Monroe, W., and Jurafsky, D. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220, 2016b. — [arXiv:1612.08220](https://arxiv.org/abs/1612.08220) find
  32. Liu, D. C. and Nocedal, J. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(1):503–528, 1989. find
  33. Liu, Y., Jiang, S., and Liao, S. Efficient approximation of cross-validation for kernel methods using Bouligand influence function. In International Conference on Machine Learning (ICML), pp. 324–332, 2014. find
  34. Martens, J. Deep learning via hessian-free optimization. In International Conference on Machine Learning (ICML), pp. 735–742, 2010. find
  35. Mei, S. and Zhu, X. The security of latent Dirichlet allocation. In Artificial Intelligence and Statistics (AISTATS), 2015a. find
  36. Mei, S. and Zhu, X. Using machine teaching to identify optimal training-set attacks on machine learners. In Association for the Advancement of Artificial Intelligence (AAAI), 2015b. find
  37. Metsis, V., Androutsopoulos, I., and Paliouras, G. Spam filtering with naive Bayes – which naive Bayes? In CEAS, volume 17, pp. 28–69, 2006. find
  38. Moosavi-Dezfooli, S., Fawzi, A., and Frossard, P. Deepfool: a simple and accurate method to fool deep neural networks. In Computer Vision and Pattern Recognition (CVPR), pp. 2574–2582, 2016. find
  39. Pearlmutter, B. A. Fast exact multiplication by the Hessian. Neural Computation, 6(1):147–160, 1994. find
  40. Pregibon, D. et al. Logistic regression diagnostics. Annals of Statistics, 9(4):705–724, 1981. find
  41. Ribeiro, M. T., Singh, S., and Guestrin, C. ”Why Should I Trust You?”: Explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining (KDD), 2016. find our card
  42. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3): 211–252, 2015. find our card
  43. Shrikumar, A., Greenside, P., and Kundaje, A. Learning important features through propagating activation differences. In International Conference on Machine Learning (ICML), 2017. find our card
  44. Simonyan, K., Vedaldi, A., and Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013. find our card
  45. Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014. — [arXiv:1412.6806](https://arxiv.org/abs/1412.6806) find
  46. Strack, B., DeShazo, J. P., Gennings, C., Olmo, J. L., Ventura, S., Cios, K. J., and Clore, J. N. Impact of HbA1c measurement on hospital readmission rates: Analysis of 70,000 clinical database patient records. BioMed Research International, 2014, 2014. find
  47. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the Inception architecture for computer vision. In Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826, 2016. find
  48. Theano D. Team. Theano: A Python framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688, 2016. find our card
  49. Thomas, W. and Cook, R. D. Assessing influence on predictions from generalized linear models. Technometrics, 32(1):59–65, 1990. van der Vaart, A. W. Asymptotic statistics. find
  50. Cambridge University Press, 1998. find
  51. Wei, B., Hu, Y., and Fung, W. Generalized leverage and its applications. Scandinavian Journal of Statistics, 25: 25–37, 1998. find
  52. Wojnowicz, M., Cruz, B., Zhao, X., Wallace, B., Wolff, M., Luan, J., and Crable, C. “Influence sketching”: Finding influential samples in large-scale regressions. arXiv preprint arXiv:1611.05923, 2016. — [arXiv:1611.05923](https://arxiv.org/abs/1611.05923) find