Elamur Research⌕
☰
home / papers / 2020 / Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?

Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?

Paper · 2020 · Peter Hase and Mohit Bansal

Summary

Many algorithms claim to make machine learning models interpretable, but there is little evidence that they help people anticipate what a model will do. Hase and Bansal test this property, called simulatability, with human-subject experiments. They use forward simulation, where users guess the model's output for unseen inputs, and counterfactual simulation, where users predict how the output shifts after an input is edited. Both tasks cover text (movie review sentiment) and tabular classification. Their design avoids earlier confounds in four ways: explained items are kept apart from test items, accuracy is measured with and without explanations, data are balanced by whether the model was right, and users must answer for every input. Five methods are compared: LIME, Anchor, Decision Boundary, Prototype and Composite. Only two results show clear gains. LIME helps on tabular data, and Prototype helps in counterfactual tests. Users' own quality ratings did not track real usefulness. The study warns against weak evaluation metrics and shows current explanation techniques leave much to improve.

Key points

  • More than 2100 user responses were gathered across forward and counterfactual simulation tasks.
  • Two existing methods (LIME, Anchor) were tested alongside two methods adapted from vision models:
    • Decision Boundary, which works by latent space traversal.
    • Prototype, which uses case-based reasoning.
  • A new Composite method merges all four. Users rated it highly, yet it did not clearly improve simulatability.
  • On the text task, no method reliably improved both forward and counterfactual simulatability, although Prototype and Composite scored best on average.
  • In forward tests, users studied 16 examples per learning phase without being able to refer back to them. Code, data and models are released at github.com/peterbhase/InterpretableNLP-ACL2020.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North Carolina at Chapel HillChapel Hill · United States37 works · 14,332 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Social Sciences · 744 works Safety Research · 221 works Software · 25 works

🧭 Topics 2 Evaluation & benchmarks · 685 works Interpretability · 58 works

Cite

Cite as Peter Hase and Mohit Bansal. 2020. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.491

@inproceedings{hase2020evaluating,
  title = {{Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?}},
  author = {Peter Hase and Mohit Bansal},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.491},
  url = {https://doi.org/10.18653/v1/2020.acl-main.491}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. find
  2. Seojin Bang, Pengtao Xie, Heewook Lee, Wei Wu, and Eric Xing. 2019. Explaining a black-box using Deep Variational Information Bottleneck Approach. arXiv:1902.06918 [cs, stat]. ArXiv: 1902.06918. — [arXiv:1902.06918](https://arxiv.org/abs/1902.06918) find
  3. Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. 2018. Do explanations make VQA models more predictable to a human? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1036–1042, Brussels, Belgium. Association… find
  4. Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. 2019. This Looks Like That: Deep Learning for Interpretable Image Recognition. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. find
  5. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. Eraser: A benchmark to evaluate rationalized nlp models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020). our card
  6. Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. arXiv:1702.08608 [cs, stat]. ArXiv: 1702.08608. our card
  7. Dheeru Dua and Casey Graff. 2017. UCI machine learning repository. find
  8. Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. CRC press. find
  9. Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining Explanations: An Overview of Interpretability of Machine Learning. The 5th IEEE International Conference on Data Science and Advanced Analytics (DSAA 2018). find
  10. Peter Hase, Chaofan Chen, Oscar Li, and Cynthia Rudin. 2019. Interpretable Image Recognition with Hierarchical Prototypes. In Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing (HCOMP-19), pages 32–40. find
  11. Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. 2019. A benchmark for interpretability methods in deep neural networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. find
  12. Amanda Hutton, Alexander Liu, and Cheryl Martin. 2012. Crowdsourcing Evaluations of Classifier Interpretability. In AAAI Spring Symposium: Wisdom of the Crowd, pages 21–26. find
  13. Shalmali Joshi, Oluwasanmi Koyejo, Been Kim, and Joydeep Ghosh. 2018. xGEMs: Generating Examplars to Explain Black-Box Models. arXiv:1806.08867 [cs, stat]. ArXiv: 1806.08867. — [arXiv:1806.08867](https://arxiv.org/abs/1806.08867) find
  14. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In Proceedings of the 35 th International Conference on Machine Learning. find
  15. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and Understanding Neural Models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681–691. Association for Computatio… our card
  16. Zachary C. Lipton. 2016. The Mythos of Model Interpretability. 2016 ICML Workshop on Human Interpretability in Machine Learning. find
  17. Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30, pages 4765–4774. our card
  18. Dong Nguyen. 2018. Comparing Automatic and Human Evaluation of Local Explanations for Text Classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1069– 1078. our card
  19. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP ’02, volume 10, pages 79–86. Association for Computational Linguistics… our card
  20. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. Knowledge Discovery and Data Mining (KDD). our card
  21. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-precision model-agnostic explanations. In AAAI Conference on Artificial Intelligence. find
  22. Cynthia Rudin. 2019. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature Machine Intelligence, 1:206–215. our card
  23. Pouya Samangouei, Ardavan Saeedi, Liam Nakagawa, and Nathan Silberman. 2018. ExplainGAN: Model Explanation via Decision Boundary Crossing Transformations. In ECCV 2018. Springer International Publishing. find
  24. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations. our card
  25. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hase-2020-evaluating-explainable-ai-which-algorithmic/