Elamur Research⌕
☰
home / papers / 2021 / Aligning Faithful Interpretations with their Social Attribution

Aligning Faithful Interpretations with their Social Attribution

Paper · 2021 · Alon Jacovi and Yoav Goldberg

Summary

The paper argues that "faithfulness," the property the NLP community usually demands of model interpretations, is too vague to ensure that explanations are useful. It uses textual highlights (extractive rationales) as a case study. Standard select-predict models first choose a highlight and then predict from it. They count as faithful, yet they can behave in ways users would never expect. One example is a selector that hides the predicted label in the shape of the mask, which the authors call "Trojan" explanations. The authors show experimentally that such failures happen in practice. Drawing on social-science work about how people explain behaviour, they separate an explanation into two parts: causal attribution, meaning the actual chain of decisions, and social attribution, meaning the intent humans read into that chain. They then define "aligned faithfulness" as a faithful causal chain that is constrained by the social role observers ascribe to it. As a remedy, they propose a predict-select-verify pipeline in which highlights can only act as evidence for the prediction already made. They implement it with contrastive highlights. The work gives interpretability research a sharper conceptual foundation.

Key points

  • Three select-predict implementations are analysed:
    • Lei et al. (2016), trained with REINFORCE;
    • Bastings et al. (2019), trained with the reparameterization trick;
    • FRESH (Jain et al., 2020), with the selector and predictor trained separately.
  • An RNN classifier that sees only the binary highlight mask, without any words, beats a random baseline on datasets such as AGNews, IMDB, Evidence Inference, 20News and Elec. This shows that label information can leak through the mask pattern.
  • The authors frame highlight usefulness around three use-cases: dispute, debug and advice.
  • Predict-select-verify keeps the original model's prediction and so cannot lower its score. Its contrastive highlights, which explain a decision against a specific alternative class, are presented as a first in NLP.
  • The paper was published in TACL vol. 9 (2021), pp. 294–310, under a CC-BY 4.0 license. Code is available at github.com/alonjacovi/aligned-highlights.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations🎓Bar-Ilan UniversityRamat Gan · Israel118 works · 21,077 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksHistory and Philosophy of Science · 18 works

Cite

Cite as Alon Jacovi and Yoav Goldberg. 2021. Aligning Faithful Interpretations with their Social Attribution. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00367

@article{jacovi2021aligning,
  title = {{Aligning Faithful Interpretations with their Social Attribution}},
  author = {Alon Jacovi and Yoav Goldberg},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00367},
  url = {https://doi.org/10.1162/tacl_a_00367}
}

References

50 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Alvarez-Melis and Tommi S. Jaakkola. 2018. On the robustness of interpretability methods. CoRR, abs/1806.08049. — [arXiv:1806.08049](https://arxiv.org/abs/1806.08049) find
  2. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. CoRR, abs/1601.01705. DOI: https://doi.org/10.18653/v1/N16-1181 ↗
  3. Joris Baan, Maartje ter Hoeve, Marlies van der Wees, Anne Schuth, and Maarten de Rijke. find
  4. Devleena Das and Sonia Chernova. 2020. Leveraging rationales to improve human task performance. In Proceedings of the 25th International Conference on Intelligent User Interfaces, IUI ’20, page 510–518. Association for Computing Machinery, New York, NY, USA. 2019. Do transformer attention heads prov… — [arXiv:1907.00570](https://arxiv.org/abs/1907.00570) find
  5. Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. 2018. Deriving machine attention from human rationales. Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, Octob… find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  7. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2019. Eraser: A benchmark to evaluate rationalized NLP models. DOI: https://doi.org/10.18653/v1/2020.acl-main.408 ↗
  8. Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963–2977. Association for Computational Linguistics, Florence, Italy. DOI: ht…
  9. Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Pai-Shun Ting, Karthikeyan Shanmugam, and Payel Das. 2018. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In NeurIPS. find
  10. Tapabrata Chakraborti, Arijit Patra, and J. Alison Noble. 2019. Contrastive algorithmic fairness: Part 1 (theory). ArXiv, abs/1905.07360. — [arXiv:1905.07360](https://arxiv.org/abs/1905.07360) find
  11. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
  12. Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. 2019. A game theoretic approach to class-wise selective rationalization. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, In Advances in Neural Information Processing Systems… find
  13. Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2020. Causalm: Causal model explanation through counterfactual language models. CoRR, abs/2005.13407.
  14. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan L. Boyd-Graber. 2018. Pathologies of neural models make interpretation difficult. Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, In Proceedings of the 2018 Conference on Empirical Methods… find
  15. Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. 2020. Invariant rationalization. CoRR, abs/2003.09772. — [arXiv:2003.09772](https://arxiv.org/abs/2003.09772) find
  16. Sheng-Hui Chen, Kayla Boggess, and Lu Feng. 2020. Towards transparent robotic planning via contrastive explanations. ArXiv, abs/2003 .07425. find
  17. Christopher Grimsley, Elijah Mayfield, and Julia R. S. Bursten. 2020. Why attention is not explanation: Surgical intervention and causal reasoning about neural models. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1780–1790. European Language Resources Association, M…
  18. Seth Chin-Parker and Julie Cantelon. 2017. Contrastive constraints guide explanation-based category learning. Cognitive Science, 41(6): 1645–1655. DOI: https://doi.org/10 .1111/cogs.12405, PMID: 27564059 find
  19. Peter Hase and Mohit Bansal. 2020. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? CoRR, abs/2005.01831. DOI: https://doi.org/10.18653/v1/2020.acl-main.491 ↗
  20. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2017. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). find
  21. Fritz Heider, American Psychological Association, and Ovid Technologies Inc. 1958. The Psychology of Interpersonal Relations. New York: Wiley ; Hillsdale, NJ: Lawrence Erlbaum. find
  22. Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. 2019, The (un)reliability of saliency methods, Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller, editors, Explainable AI: I… — [doi:10.1007/978-3-030-28954](https://doi.org/10.1007/978-3-030-28954) find
  23. Fritz Heider and Marianne Simmel. 1944. An experimental study of apparent behavior. The American Journal of Psychology, 57(2): 243–259. DOI: https://doi.org/10 .2307/1416950 find
  24. Bernease Herman. 2017. The promise and peril of human evaluation for model interpretability. CoRR, abs/1711.07414. Withdrawn. — [arXiv:1711.07414](https://arxiv.org/abs/1711.07414) find
  25. Germund Hesslow. 1988, The problem of causal selection, Denis J. Hilton, editor, Contemporary Science and Natural Explanation: Commonsense Conceptions of Causality, New York University Press. find
  26. Diederik P. Kingma and Max Welling. 2014. Auto-encoding variational bayes. Yoshua Bengio and Yann LeCun, editors, In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
  27. Hilton, Denis J. 1988. Logic and causal attribution. In Contemporary Science and Natural Explanation: Commonsense Conceptions of Causality. 33–65, New York University Press. find
  28. I. Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, and Sorelle A. Friedler. 2020. Problems with shapley-value-based explanations as feature importance measures. CoRR, abs/2002.11097. — [arXiv:2002.11097](https://arxiv.org/abs/2002.11097) find
  29. Denis J. Hilton, 1966 Mandel, David R., and Patrizia Catellani. 2005. The Psychology of Counterfactual Thinking, London ; New York : Routledge. Includes bibliographical references (p. [217]-244) and indexes. find
  30. Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2016. Rationalizing neural predictions. CoRR, abs/1606.04155. DOI: https://doi.org/10.18653/v1/D16-1011 ↗
  31. Peter Lipton. 1990. Contrastive explanation. Royal Institute of Philosophy Supplements, 27:247–266. DOI: https://doi.org/10 .1017/S1358246100005130 find
  32. Denis J. Hilton and Ben R. Slugoski. 1986. Knowledge-based causal attribution: The abnormal conditions focus model. Psychological Review, 93(1):75. DOI: https://doi.org/10.1037/0033-295X.93.1.75 — [doi:10.1037/0033-295x.93.1.75](https://doi.org/10.1037/0033-295x.93.1.75) ↗
  33. Zachary C. Lipton. 2018. The mythos of model interpretability. Communications of ACM, 61(10): 36–43. DOI: https://doi.org/10.1145/3233231 ↗
  34. Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? CoRR, abs/2004.03685. DOI: https://doi.org/10.18653/v1/2020.acl-main.386 ↗
  35. Tania Lombrozo. 2006. The structure and function of explanations. Trends in Cognitive Sciences, 10(10):464–470. DOI: https://doi.org/10.1016/j.tics.2006.08.004, PMID: 16942895 ↗
  36. Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. 2020. Learning to faithfully rationalize by construction. CoRR, abs/ 2005.00115. DOI: https://doi.org/10 .18653/v1/2020.acl-main.409
  37. Scott M. Lundberg and Su-In Lee. 2017, A unified approach to interpreting model predictions, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, Curran Associates, Inc., pages 4765–4774. https://do…
  38. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2931–2951.
  39. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684. Association for Computational Linguistics, Florence, Italy. DOI: https://doi.org /… ↗
  40. Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. 2020. Obtaining faithful interpretations from compositional neural networks. CoRR, abs/2005.00724. DOI: https://doi.org/10.18653/v1/2020 .acl-main.495 ↗
  41. Ann Mcgill and Jill Klein. 1993. Contrastive and counterfactual thinking in causal judgment. Journal of Personality and Social Psychology, 64:897–905. DOI: https://doi.org/10 .1037/0022-3514.64.6.897 find
  42. Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267:1–38. DOI: https://doi.org/10.1016/j.artint.2018.07 .007 ↗
  43. Tim Miller. 2020. Contrastive explanation: A structural-model approach. find
  44. Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2019. Learning to deceive with attention-based explanations. CoRR, abs/1909.07913. DOI: https://doi.org/10.18653/v1/2020.acl-main .432 ↗
  45. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. ‘‘Why Should I Trust You?’’: Explaining the predictions of any classifier. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 1135–1144. ACM, New York, NY, USA. DOI: https…
  46. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin…
  47. Joseph Jay Williams, Tania Lombrozo, and Bob Rehder. 2013. The hazards of explanation: Overgeneralization in the face of exceptions. Journal of Experimental Psychology: General, 142(4):1006. DOI: https://doi.org/10 .1037/a0030996, PMID: 23294346 find
  48. Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256. DOI: https://doi.org/10.1007/BF00992696 ↗
  49. Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206–215.
  50. DOI: Omar Zaidan, Jason Eisner, and Christine D. Piatko. 2007. Using ‘‘annotator rationales’’ to find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jacovi-2021-aligning-faithful-interpretations-their-social/