Elamur Research
☰
home / papers / 2016 / "Why Should I Trust You?": Explaining the Predictions of Any Classifier

"Why Should I Trust You?": Explaining the Predictions of Any Classifier

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1602.04938

Summary

Machine learning models often act as black boxes, which makes it hard to decide whether to act on a single prediction or to deploy a whole model. The authors introduce LIME (Local Interpretable Model-agnostic Explanations). It treats any classifier or regressor as a black box. To explain one prediction, it perturbs the instance in an interpretable representation, such as the presence or absence of words or image superpixels. It then fits a simple, sparse linear surrogate model, weighted by how close each perturbed sample is to the original. A second method, SP-LIME, frames the choice of a small, non-redundant set of explained examples as submodular optimization, which gives a global picture of the model. The authors test both methods on text classifiers (including random forests) and on image neural networks, in simulated studies and with human subjects. Explanations helped people, including non-experts, choose the classifier that generalizes better, find spurious features caused by data leakage or dataset shift, and fix weak models. This matters because explanations offer a check on trust that goes beyond validation accuracy.

Key points

  • LIME balances two goals: local fidelity, meaning it matches the model near the instance being explained, and interpretability, meaning it keeps the explanation short. It needs no access to the model's internals.
  • SP-LIME uses greedy submodular selection to choose representative instances, so users don't have to inspect a large dataset at random.
  • On the 20 newsgroups "Christianity vs Atheism" task, a classifier with higher validation accuracy turned out to depend on meaningless artifacts such as header words. Non-expert users who removed features based on LIME explanations produced a classifier that generalized better.
  • Crowd workers given SP-LIME explanations picked the model that generalizes better about 89% of the time. This figure comes from my knowledge of the paper; the provided excerpt omitted the results section.
  • The authors also explain image predictions from a neural network (Google's Inception). Planned future work includes other explanation families such as decision trees, a pick step for images, and faster explanations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

816 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 795 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1602.04938 · arXiv 1602.04938

@misc{ribeiro2016why,
  title = {{"Why Should I Trust You?": Explaining the Predictions of Any Classifier}},
  author = {Ribeiro, Marco Tulio and Singh, Sameer and Guestrin, Carlos},
  year = {2016},
  doi = {10.48550/arxiv.1602.04938},
  eprint = {1602.04938},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1602.04938}
}

References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Amershi, M. Chickering, S. M. Drucker, B. Lee, P. Simard, and J. Suh. Modeltracker: Redesigning performance analysis tools for machine learning. In Human Factors in Computing Systems (CHI), 2015. find
  2. D. Baehrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, and K.-R. Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 11, 2010. find
  3. A. Bansal, A. Farhadi, and D. Parikh. Towards transparent systems: Semantic characterization of failure modes. In European Conference on Computer Vision (ECCV), 2014. find
  4. J. Blitzer, M. Dredze, and F. Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Association for Computational Linguistics (ACL), 2007. find
  5. J. Q. Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset Shift in Machine Learning. MIT, 2009. find
  6. R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, and N. Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Knowledge Discovery and Data Mining (KDD), 2015. find
  7. M. W. Craven and J. W. Shavlik. Extracting tree-structured representations of trained networks. Neural information processing systems (NIPS), pages 24–30, 1996. find
  8. M. T. Dzindolet, S. A. Peterson, R. A. Pomranky, L. G. Pierce, and H. P. Beck. The role of trust in automation reliance. Int. J. Hum.-Comput. Stud., 58(6), 2003. find
  9. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of Statistics, 32:407–499, 2004. find
  10. U. Feige. A threshold of ln n for approximating set cover. J. ACM, 45(4), July 1998. find
  11. A. Groce, T. Kulesza, C. Zhang, S. Shamasunder, M. Burnett, W.-K. Wong, S. Stumpf, S. Das, A. Shinsel, F. Bice, and K. McIntosh. You are the only possible oracle: Effective test selection for end users of interactive machine learning systems. IEEE Trans. Softw. Eng., 40(3), 2014. find
  12. J. L. Herlocker, J. A. Konstan, and J. Riedl. Explaining collaborative filtering recommendations. In Conference on Computer Supported Cooperative Work (CSCW), 2000. find
  13. A. Karpathy and F. Li. Deep visual-semantic alignments for generating image descriptions. In Computer Vision and Pattern Recognition (CVPR), 2015. find our card
  14. S. Kaufman, S. Rosset, and C. Perlich. Leakage in data mining: Formulation, detection, and avoidance. In Knowledge Discovery and Data Mining (KDD), 2011. find
  15. A. Krause and D. Golovin. Submodular function maximization. In Tractability: Practical Approaches to Hard Problems. Cambridge University Press, February 2014. find
  16. T. Kulesza, M. Burnett, W.-K. Wong, and S. Stumpf. Principles of explanatory debugging to personalize interactive machine learning. In Intelligent User Interfaces (IUI), 2015. find
  17. B. Letham, C. Rudin, T. H. McCormick, and D. Madigan. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. Annals of Applied Statistics, 2015. find
  18. D. Martens and F. Provost. Explaining data-driven document classifications. MIS Q., 38(1), 2014. find
  19. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Neural Information Processing Systems (NIPS). 2013. find
  20. K. Patel, J. Fogarty, J. A. Landay, and B. Harrison. Investigating statistical machine learning as a tool for software development. In Human Factors in Computing Systems (CHI), 2008. find
  21. K. Patel, N. Bancroft, S. M. Drucker, J. Fogarty, A. J. Ko, and J. Landay. Gestalt: Integrated support for implementation and analysis in machine learning. In User Interface Software and Technology (UIST), 2010. find
  22. I. Sanchez, T. Rocktaschel, S. Riedel, and S. Singh. Towards extracting faithful and descriptive representations of latent variable models. In AAAI Spring Syposium on Knowledge Representation and Reasoning (KRR): Integrating Symbolic and Neural Approaches, 2015. find
  23. D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, and J.-F. Crespo. Hidden technical debt in machine learning systems. In Neural Information Processing Systems (NIPS). 2015. find
  24. E. Strumbelj and I. Kononenko. An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 11, 2010. find
  25. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Computer Vision and Pattern Recognition (CVPR), 2015. find
  26. B. Ustun and C. Rudin. Supersparse linear integer models for optimized medical scoring systems. Machine Learning, 2015. find
  27. F. Wang and C. Rudin. Falling rule lists. In Artificial Intelligence and Statistics (AISTATS), 2015. find
  28. K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning (ICML), 2015. find our card
  29. P. Zhang, J. Wang, A. Farhadi, M. Hebert, and D. Parikh. Predicting failures of vision systems. In Computer Vision and Pattern Recognition (CVPR), 2014. find