Elamur Research⌕
☰
home / papers / 2019 / AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models

AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models

Paper · 2019 · Eric Wallace, Jens Tuyls, Junlin Wang et al.

Summary

Neural NLP models are accurate but opaque. Most available explanation code is built for computer vision, tied to a single model or task, or covers only a few techniques, so practitioners rarely use it and researchers struggle to test their methods across systems. The authors present AllenNLP Interpret, an open-source extension of AllenNLP with three parts: ready-made instance-level explanation methods, model- and task-agnostic primitives (chiefly input gradients) for building new methods, and reusable web components for interactive display.

The toolkit focuses on gradient-based techniques, namely three saliency-map variants and two kinds of input-perturbation attacks, because these apply to any differentiable model. The only model-specific code is a short function that turns a model's own prediction into labeled instances. Gradients are then captured through a hook on the embedding layer.

Live demos run five methods on six tasks, including BERT masked language modeling and BiDAF reading comprehension. They expose biases, decision rules and errors. The work makes explaining arbitrary NLP models much cheaper.

Key points

  • Saliency methods. The toolkit implements Vanilla Gradient, Integrated Gradients (with an all-zero embedding baseline) and SmoothGrad (Gaussian noise added to embeddings). Each token's gradient is collapsed to one score using the L2 norm.
  • Attack methods.
    • HotFlip, in its original untargeted form and in a new targeted variant that pushes the model toward a chosen output.
    • Input Reduction, which repeatedly deletes the lowest-gradient word while keeping the prediction unchanged.
  • Supported tasks and models.
    • Reading comprehension on SQuAD and DROP, using BiDAF and NAQANet.
    • Masked LM with PyTorch Transformers models such as BERT and RoBERTa.
    • Text classification and textual entailment with BiLSTM and self-attention classifiers.
    • NER and coreference resolution.
  • Example findings from the demos.
    • A SQuAD model relies on question–passage word overlap.
    • An entailment model predicts contradiction when it sees "politics".
    • An NER model links "in downtown" to the Location tag.
    • A sentiment model is misled by "tony hawk style".
  • Extension effort.
    • Adding SmoothGrad took about ten lines on top of vanilla gradient, about five lines of demo back-end code, and a single-line front-end call.
    • The labeled-instance function for a classifier needs about three lines.

Interesting ideas

  • I1. Explain predictions rather than gold labels. The model's own output is used as the target when computing the loss. This lets the same machinery explain any prediction, including on unlabeled data.
  • I2. Split structured outputs into separate instances. Each NER tag or coreference cluster becomes its own instance, so every part of the output gets its own explanation. This is useful for any sequence-labeling or parsing system.
  • I3. Embedding-agnostic gradients via a hook. A backward hook on the shared token-embedding interface captures gradients regardless of whether the model uses word vectors, ELMo or BERT. This is a cheap way to add introspection to an existing framework without touching individual models.
  • I4. A substitute vocabulary matrix for contextual embeddings. Vocabulary items are passed through the model's last context-independent layer (for example, ELMo's Char-CNN), and the results serve as a searchable matrix for word-swap attacks. Gradients still flow through the contextual layers, so this transfers to other discrete-search methods on contextual models.
  • I5. Batch mode alongside the visual demo. The tool can also run offline over many examples, which allows aggregate statistics such as typical reduced-input length rather than one-off anecdotes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations🎓University of California, IrvineIrvine · United States29 works · 36,049 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner and Sameer Singh. 2019. AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-3002

@inproceedings{wallace2019allennlp,
  title = {{AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models}},
  author = {Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner and Sameer Singh},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations},
  year = {2019},
  doi = {10.18653/v1/d19-3002},
  url = {https://doi.org/10.18653/v1/d19-3002}
}

References

22 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. our card
  2. Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, et al. 2016. Technical report on the CleverHans v2.1.0 adversarial examples library. arXiv:1610.00768. — [arXiv:1610.00768](https://arxiv.org/abs/1610.00768) find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL. our card
  4. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. our card
  5. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. our card
  6. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL. our card
  7. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP. our card
  8. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should I trust you?: Explaining the predictions of any classifier. In KDD. our card
  9. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In ACL. our card
  10. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR. our card
  11. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In EMNLP. find
  12. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR. our card
  13. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In ACL Workshop for NLP Open Source Software. our card
  14. Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. 2017. SmoothGrad: removing noise by adding noise. In ICML Workshop on Visualization for Deep Learning. find
  15. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In NAACL. find
  16. Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexander M Rush. 2019. Seq2Seq-Vis: A visual debugging tool for sequence-to-sequence models. IEEE TVCG. find
  17. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2016. Visualizing and understanding recurrent networks. In ICLR Workshop Track. our card
  18. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In ICML. find
  19. Gyeongbok Lee, Sungdong Kim, and Seung-won Hwang. 2019. QADiver: Interactive framework for diagnosing QA models. In AAAI Demonstrations. find
  20. Jesse Vig. 2019. Visualizing attention in transformer-based language models. arXiv:1904.02679. — [arXiv:1904.02679](https://arxiv.org/abs/1904.02679) find
  21. Shusen Liu, Tao Li, Zhimin Li, Vivek Srikumar, Valerio Pascucci, and Peer-Timo Bremer. 2018. Visual interrogation of attention-based models for natural language inference and machine comprehension. In EMNLP. find
  22. Eric Wallace, Shi Feng, and Jordan Boyd-Graber. 2018. Interpreting neural networks with nearest neighbors. In EMNLP 2018 Blackbox NLP Workshop. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wallace-2019-allennlp-interpret-framework-explaining-predictions/