Elamur Research⌕
☰
home / papers / 2020 / Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions

Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions

Summary

Popular ways of explaining neural text classifiers, such as gradient saliency or attention heatmaps, score individual input tokens. The authors argue that this works poorly for reasoning-heavy tasks, where a decision depends on how several words or spans interact. As an NLP alternative, they evaluate the influence-function technique proposed by Koh and Liang (2017). Influence functions explain a test prediction by ranking the training examples that most shaped it.

Classifiers built on BERT-Base were trained for sentiment (SST-2) and for entailment (MNLI), and explanations were evaluated on HANS diagnostic examples. The authors first confirm that the influence approximation is reliable enough for deep transformers. They then compare it with "gradient × input" saliency. The two methods mostly agree on lexicon-driven sentiment but diverge on NLI, which suggests that example-based explanations suit inference-style tasks better. Finally, the paper introduces an influence-based quantitative measure of how much annotation artifacts, such as lexical overlap or negation, drive predictions. This gives practitioners a way to debug datasets, not just individual model outputs.

Key points

  • Model setup. BERT-Base was fine-tuned with the lower 8 of 12 layers frozen. Influence was estimated with LiSSA plus a Hessian damping term to handle non-convexity.
  • Sentiment and NLI results. The SST-2 model was trained on 10k examples and reached 89.6% on the 872-example dev set. The MNLI model was trained on 10k sampled pairs, with neutral and contradiction merged into a single non-entailment label, and reached 84.6% on the 9,815-example dev set.
  • Evaluation samples. Explanations were examined for 50 sampled SST-2 dev examples and 30 HANS items covering all 30 heuristic sub-categories.
  • Cost. Influence functions are expensive: scoring 10k training examples for a single test input took about 10 minutes on an RTX 2080 Ti.
  • Code. The implementation is released at github.com/xhan77/influence-function-analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

76 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 75 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.492

@inproceedings{han2020explaining,
  title = {{Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions}},
  author = {Han, Xiaochuang and Wallace, Byron C. and Tsvetkov, Yulia},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.492},
  url = {https://doi.org/10.18653/v1/2020.acl-main.492}
}

References

39 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Naman Agarwal, Brian Bullins, and Elad Hazan. 2017. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research (JMLR), 18:116:1–116:40. find
  2. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72. find our card
  3. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations. In Proc. NeurIPS. find our card
  4. Dallas Card, Michael Zhang, and Noah A. Smith. 2019. Deep weighted averaging classifiers. In FAT*. find
  5. Rich Caruana, Hooshang Kangarloo, John David N. Dionisio, Usha S. Sinha, and David B. Johnson. 1999. Case-based explanation of non-case-based learning methods. Proc. AMIA Symposium, pages 212–5. find
  6. Danilo Croce, Daniele Rossini, and Roberto Basili. 2019. Auditing deep learning processes through kernel-based explanatory models. In Proc. EMNLP. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT. find
  8. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. find our card
  9. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proc. ACL. find our card
  10. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proc. ACL. find our card
  11. Shi Feng, Eric Wallace, Alvin Grissom, Mohit Iyyer, Pedro Rodriguez, and Jordan L. Boyd-Graber. 2018. Pathologies of neural models make interpretation difficult. In Proc. EMNLP. find
  12. Reid Pryzant, Kelly Shen, Dan Jurafsky, and Stefan Wagner. 2018. Deconfounded lexicon induction for interpretable social science. In NAACL-HLT. find
  13. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proc. NAACL-HLT. find
  14. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361. find our card
  15. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proc. EMNLP, pages 2733–2743. find
  16. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “why should i trust you?”: Explaining the predictions of any classifier. In Proc. HLT-NAACL (System Demonstrations). find our card
  17. Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? ArXiv, abs/2004.03685. find our card
  18. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proc. ACL. find
  19. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proc. NAACL-HLT. find our card
  20. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In Proc. ICML. find our card
  21. Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. 2020. Learning to faithfully rationalize by construction. In Proc. ACL. find our card
  22. Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proc. ICML. find our card
  23. Sachin Kumar, Shuly Wintner, Noah A. Smith, and Yulia Tsvetkov. 2019. Topics to avoid: Demoting latent confounds in text classification. In Proc. EMNLP, pages 4151–4161. find
  24. Virgile Landeiro, Tuan Tran, and Aron Culotta. 2019. Discovering and controlling for latent confounds in text classification using adversarial domain adaptation. In Proc. SIAM International Conference on Data Mining, pages 298–305. find
  25. Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2016. Rationalizing neural predictions. In Proc. EMNLP. find our card
  26. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations. find our card
  27. Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. 2017. Smooth-Grad: removing noise by adding noise. ArXiv, abs/1706.03825. — [arXiv:1706.03825](https://arxiv.org/abs/1706.03825) find
  28. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. EMNLP. find
  29. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proc. ICML. find
  30. Jiwei Li, Xinlei Chen, Eduard H. Hovy, and Dan Jurafsky. 2016a. Visualizing and understanding neural models in NLP. In Proc. HLT-NAACL. find our card
  31. Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh. 2019. AllenNLP interpret: A framework for explaining predictions of NLP models. In Proc. EMNLP (System Demonstrations), pages 7–12. find
  32. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b. Understanding neural networks through representation erasure. ArXiv, abs/1612.08220. — [arXiv:1612.08220](https://arxiv.org/abs/1612.08220) find
  33. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proc. EMNLP, pages 11–20. find our card
  34. Zachary Chase Lipton. 2018. The mythos of model interpretability. Commun. ACM, 61:36–43. find
  35. Hui Liu, Qingyu Yin, and William Yang Wang. 2019a. Towards explainable NLP: A generative explanation framework for text classification. In Proc. ACL. find
  36. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019b. Linguistic knowledge and transferability of contextual representations. In Proc. NAACL-HLT. find our card
  37. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. NAACL-HLT. find
  38. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  39. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proc. ICML. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/han-2020-explaining-black-box-predictions-unveiling/