Elamur Research
☰
home / papers / 2017 / A Unified Approach to Interpreting Model Predictions

A Unified Approach to Interpreting Model Predictions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1705.07874

Summary

Highly accurate predictors such as deep networks and ensembles are hard to interpret. Many explanation techniques have been proposed for them, but it has been unclear how these techniques relate or when to prefer one. Lundberg and Lee treat an explanation as a simple model that locally mimics the original one. They introduce a family, termed additive feature attribution methods, in which a prediction is decomposed into a base value plus contributions from each feature. LIME, DeepLIFT, layer-wise relevance propagation, Shapley regression values, Shapley sampling values and Quantitative Input Influence all fall into this class. Using cooperative game theory, the authors prove that exactly one attribution in the class satisfies local accuracy, missingness and consistency. That attribution is the set of Shapley values of a conditional expectation of the model, which they call SHAP values. They also derive practical estimators, notably Kernel SHAP and Deep SHAP. Experiments show these estimators are more sample-efficient, agree better with human judgments and separate classes better on MNIST than earlier methods. The result gives model interpretation a principled, shared foundation.

Key points

  • Six existing methods are shown to be special cases of one explanation form: a linear function over binary "feature present/absent" variables.
  • Theorem 1 says Shapley values are the only attributions in this class that satisfy all three properties. Methods not based on them can break local accuracy and/or consistency.
  • Kernel SHAP links Shapley values to weighted linear regression. It needs fewer model evaluations than classic Shapley sampling, especially when regularization is added. Deep SHAP is a compositional approximation inspired by DeepLIFT.
  • In Amazon Mechanical Turk studies (30 and 52 participants), people's credit assignments matched SHAP more closely than LIME or the original DeepLIFT. The tasks were a symptom-based sickness score and a max-based profit split.
  • On an MNIST CNN (2 conv + 2 dense layers + 10-way softmax), the authors masked 20% of pixels to turn an 8 into a 3, averaged over 20 images. Methods closer to SHAP values produced larger log-odds changes. The code is released at github.com/slundberg/shap.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

616 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 606 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1705.07874 · arXiv 1705.07874

@misc{lundberg2017unified,
  title = {{A Unified Approach to Interpreting Model Predictions}},
  author = {Lundberg, Scott and Lee, Su-In},
  year = {2017},
  doi = {10.48550/arxiv.1705.07874},
  eprint = {1705.07874},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.07874}
}

References

10 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sebastian Bach et al. “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation”. In: PloS One 10.7 (2015), e0130140. find
  2. A Charnes et al. “Extremal principle solutions of games in characteristic function form: core, Chebychev and Shapley value generalizations”. In: Econometrics of Planning and Efficiency 11 (1988), pp. 123–133. find
  3. Anupam Datta, Shayak Sen, and Yair Zick. “Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems”. In: Security and Privacy (SP), 2016 IEEE Symposium on. IEEE. 2016, pp. 598–617. find
  4. Stan Lipovetsky and Michael Conklin. “Analysis of regression in game theory approach”. In: Applied Stochastic Models in Business and Industry 17.4 (2001), pp. 319–330. find
  5. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “Why should i trust you?: Explaining the predictions of any classifier”. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM. 2016, pp. 1135–1144. find our card
  6. Lloyd S Shapley. “A value for n-person games”. In: Contributions to the Theory of Games 2.28 (1953), pp. 307–317. find
  7. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. “Learning Important Features Through Propagating Activation Differences”. In: arXiv preprint arXiv:1704.02685 (2017). find our card
  8. Avanti Shrikumar et al. “Not Just a Black Box: Learning Important Features Through Propagating Activation Differences”. In: arXiv preprint arXiv:1605.01713 (2016). find our card
  9. Erik Štrumbelj and Igor Kononenko. “Explaining prediction models and individual predictions with feature contributions”. In: Knowledge and information systems 41.3 (2014), pp. 647–665. find
  10. H Peyton Young. “Monotonic solutions of cooperative games”. In: International Journal of Game Theory 14.2 (1985), pp. 65–72. find