Elamur Research
☰
home / papers / 2017 / Learning Important Features Through Propagating Activation Differences

Learning Important Features Through Propagating Activation Differences

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1704.02685

Summary

Neural networks are often seen as opaque, which limits their use where explanations matter. This paper introduces DeepLIFT (Deep Learning Important FeaTures), which attributes a network's prediction for a given input to individual input features. Instead of relying on local gradients, DeepLIFT measures how far each neuron's activation departs from its value on a chosen "reference" input. It then distributes the change in the output across these departures so that the attributions sum exactly to the output change. A chain rule over "multipliers" (finite-difference analogues of partial derivatives) lets the scores be computed in one backward pass. An optional rule, RevealCancel, handles positive and negative contributions separately. This exposes interactions that other methods overlook. The authors test it on MNIST and on simulated DNA sequences containing GATA1 and TAL1 motifs. Compared with gradients, gradient×input and Guided Backprop, DeepLIFT handles saturated units, avoids jumps caused by thresholds and bias terms, and better reveals motif cooperativity. This matters for trustworthy interpretation, for example in genomics.

Key points

  • Core property ("summation-to-delta"): the contribution scores of the inputs add up to the output's difference from its reference value. Scores can therefore be nonzero even where the gradient vanishes, as with saturation.
  • Choice of reference: the reference is problem-specific, and gradient×input implicitly assumes an all-zeros reference.
    • MNIST: all-zeros.
    • DNA: background base frequencies (A/C/G/T = 0.3/0.2/0.2/0.3) or shuffled copies of each sequence.
    • CIFAR10: a blurred copy of the image highlights object outlines better than zeros.
  • Genomics experiment: a CNN (two conv layers, global average pooling, one dense layer) was trained on 3 binary tasks: both motifs present, GATA1 present, and TAL1 present. Each of the four motif combinations made up a quarter of the data. Methods were scored on the top 5 log-odds motif matches per test sequence.
  • Variants compared: DeepLIFT-Rescale, DeepLIFT-RevealCancel, and DeepLIFT-fc-RC-conv-RS (Rescale on conv layers, RevealCancel on the dense layer). On DNA data the mixed variant gave the least noise and recovered TAL1 relevance in the joint task, which other methods sometimes missed.
  • Weaknesses of competing methods:
    • Guided Backprop×input failed to detect cooperativity between the two motifs.
    • Both Guided Backprop×input and gradient×input inflated the importance of weak motif matches.
    • Integrated Gradients is costlier to compute.
    • Open questions: extending DeepLIFT to RNNs, learning references from data, and handling max operations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

178 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 175 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1704.02685 · arXiv 1704.02685

@misc{shrikumar2017learning,
  title = {{Learning Important Features Through Propagating Activation Differences}},
  author = {Shrikumar, Avanti and Greenside, Peyton and Kundaje, Anshul},
  year = {2017},
  doi = {10.48550/arxiv.1704.02685},
  eprint = {1704.02685},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1704.02685}
}

References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bach, Sebastian, Binder, Alexander, Montavon, Grégoire, Klauschen, Frederick, Müller, Klaus-Robert, and Samek, Wojciech. On Pixel-Wise explanations for Non-Linear classifier decisions by Layer-Wise relevance propagation. PLoS One, 10(7):e0130140, 10 July 2015. find
  2. Chollet, Franois. keras. https://github.com/fchollet/keras, 2015. ↗
  3. Kheradpour, Pouya and Kellis, Manolis. Systematic discovery and characterization of regulatory motifs in encode tf binding experiments. Nucleic acids research, 42 (5):2976–2987, 2014. find
  4. Kindermans, Pieter-Jan, Schtt, Kristof, Mller, Klaus-Robert, and Dhne, Sven. Investigating the influence of noise and distractors on the interpretation of neural networks. CoRR, abs/1611.07270, 2016. URL https: //arxiv.org/abs/1611.07270. — [arXiv:1611.07270](https://arxiv.org/abs/1611.07270) find
  5. LeCun, Yann, Cortes, Corinna, and Burges, Christopher J.C. The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/, 1999. ↗
  6. Lundberg, Scott and Lee, Su-In. An unexpected unity among methods for interpreting model predictions. CoRR, abs/1611.07478, 2016. URL http://arxiv. org/abs/1611.07478. — [arXiv:1611.07478](https://arxiv.org/abs/1611.07478) find
  7. Selvaraju, Ramprasaath R., Das, Abhishek, Vedantam, Ramakrishna, Cogswell, Michael, Parikh, Devi, and Batra, Dhruv. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. CoRR, abs/1610.02391, 2016. URL http: //arxiv.org/abs/1610.02391. find our card
  8. Shrikumar, Avanti, Greenside, Peyton, Shcherbina, Anna, and Kundaje, Anshul. Not just a black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713, 2016. — [arXiv:1605.01713](https://arxiv.org/abs/1605.01713) find
  9. Simonyan, Karen, Vedaldi, Andrea, and Zisserman, Andrew. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013. find our card
  10. Springenberg, Jost Tobias, Dosovitskiy, Alexey, Brox, Thomas, and Riedmiller, Martin A. Striving for simplicity: The all convolutional net. CoRR, abs/1412.6806, 6806. 2014. URL http://arxiv.org/abs/1412. — [arXiv:1412.6806](https://arxiv.org/abs/1412.6806) ↗
  11. Sundararajan, Mukund, Taly, Ankur, and Yan, Qiqi. Gradients of counterfactuals. CoRR, abs/1611.02639, 2016. URL http://arxiv.org/abs/1611.02639. — [arXiv:1611.02639](https://arxiv.org/abs/1611.02639) ↗
  12. Zeiler, Matthew D. and Fergus, Rob. Visualizing and understanding convolutional networks. CoRR, abs/1311.2901, 2013. URL http://arxiv.org/abs/1311.2901. ↗ our card
  13. Zhou, Jian and Troyanskaya, Olga G. Predicting effects of noncoding variants with deep learning-based sequence model. Nat Methods, 12:931–4, 2015 Oct 2015. ISSN 1548-7105. doi: 10.1038/nmeth.3547. — [doi:10.1038/nmeth.3547](https://doi.org/10.1038/nmeth.3547) find
  14. Zintgraf, Luisa M, Cohen, Taco S, Adel, Tameem, and Welling, Max. Visualizing deep neural network decisions: Prediction difference analysis. ICLR, 2017. URL https://openreview.net/pdf?id=BJ5UeU9xx.7. Acknowledgements We thank Anna Shcherbina for early experiments applying DeepLIFT to image data and… ↗