Elamur Research
☰
home / papers / 2017 / Towards A Rigorous Science of Interpretable Machine Learning

Towards A Rigorous Science of Interpretable Machine Learning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1702.08608

Summary

Machine learning systems are increasingly deployed where goals like safety, fairness, or a legal right to explanation matter, yet these goals are hard to specify completely. Interpretability is often used as a fallback check, but the field has no agreed meaning or measurement for it. Most papers either show usefulness in some application or simply declare a model class (e.g., sparse linear models, rule lists) interpretable. This position paper by Doshi-Velez and Kim introduces no new data or models. Instead, it proposes a conceptual framework. It treats interpretability as the capacity to make a model's behaviour understandable to a human. It argues that explanations are needed when a problem's formalization is incomplete, which differs from uncertainty that can be quantified. The central contribution is a three-level taxonomy of evaluation, ordered by cost and specificity, together with hypothesized latent dimensions of interpretability. The work matters because it gives researchers a shared vocabulary for comparing methods and for matching the type of evaluation to the type of claim.

Key points

  • Evaluation taxonomy: application-grounded evaluation uses real humans on real tasks, such as doctors making diagnoses. Human-grounded evaluation uses real humans on simplified tasks. Functionally-grounded evaluation uses no humans and relies on proxy tasks. Moving down the list, evaluations become cheaper and less specific.
  • Why incompleteness matters: the authors illustrate gaps in problem formalization with five scenarios: scientific understanding, safety, ethics, mismatched objectives, and multi-objective trade-offs.
  • Hypothesized dimensions: task-related factors include global vs. local scope, severity of incompleteness, user expertise, and time constraints. Method-related factors include the form and number of "cognitive chunks," compositionality, monotonicity, and the handling of uncertainty.
  • Recommendations: the evaluation should match the claim. Researchers should state how the problem is incomplete, at which evaluation level they work, and which task and method factors they examine.
  • Context: the paper cites the upcoming EU "right to explanation" (2018) and notes that Google Scholar lists over 20,000 interpretability publications from the preceding five years.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

200 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 192 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1702.08608 · arXiv 1702.08608

@misc{doshivelez2017rigorous,
  title = {{Towards A Rigorous Science of Interpretable Machine Learning}},
  author = {Doshi-Velez, Finale and Kim, Been},
  year = {2017},
  doi = {10.48550/arxiv.1702.08608},
  eprint = {1702.08608},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1702.08608}
}

References

51 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. find our card
  2. Pedro Antunes, Valeria Herskovic, Sergio F Ochoa, and Jose A Pino. Structuring dimensions for collaborative systems evaluation. ACM Computing Surveys, 2012. find
  3. William Bechtel and Adele Abrahamsen. Explanation: A mechanist alternative. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences, 2005. find
  4. Catherine Blake and Christopher J Merz. {UCI} repository of machine learning databases. 1998. find
  5. Nick Bostrom and Eliezer Yudkowsky. The ethics of artificial intelligence. The Cambridge Handbook of Artificial Intelligence, 2014. find
  6. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. find
  7. Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2006. find
  8. Samuel Carton, Jennifer Helsby, Kenneth Joseph, Ayesha Mahmud, Youngsoo Park, Joe Walsh, Crystal Cody, CPT Estella Patterson, Lauren Haynes, and Rayid Ghani. Identifying police officers at risk of adverse events. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 201… find
  9. Jonathan Chang, Jordan L Boyd-Graber, Sean Gerrish, Chong Wang, and David M Blei. Reading tea leaves: How humans interpret topic models. In NIPS, 2009. find our card
  10. Nick Chater and Mike Oaksford. Speculations on human causal learning and reasoning. Information sampling and adaptive cognition, 2006. find
  11. Finale Doshi-Velez, Yaorong Ge, and Isaac Kohane. Comorbidity clusters in autism spectrum disorders: an electronic health record time-series analysis. Pediatrics, 133(1):e54–e63, 2014. find
  12. Finale Doshi-Velez, Byron Wallace, and Ryan Adams. Graph-sparse lda: a topic model with structured sparsity. Association for the Advancement of Artificial Intelligence, 2015. find
  13. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science Conference. ACM, 2012. find
  14. Alex Freitas. Comprehensible classification models: a position paper. ACM SIGKDD Explorations, 2014. find
  15. Vikas K Garg and Adam Tauman Kalai. Meta-unsupervised-learning: A supervised approach to unsupervised learning. arXiv preprint arXiv:1612.09030, 2016. — [arXiv:1612.09030](https://arxiv.org/abs/1612.09030) find
  16. Stuart Glennan. Rethinking mechanistic explanation. Philosophy of science, 2002. find
  17. Bryce Goodman and Seth Flaxman. European union regulations on algorithmic decision-making and a” right to explanation”. arXiv preprint arXiv:1606.08813, 2016. — [arXiv:1606.08813](https://arxiv.org/abs/1606.08813) find
  18. Maya Gupta, Andrew Cotter, Jan Pfeifer, Konstantin Voevodski, Kevin Canini, Alexander Mangylov, Wojciech Moczydlowski, and Alexander Van Esbroeck. Monotonic calibrated interpolated look-up tables. Journal of Machine Learning Research, 2016. find
  19. Sean Hamill. CMU computer won poker battle over humans by statistically significant margin. http://www.post-gazette.com/business/tech-news/2017/01/31/CMU-computer-won-poker-battle-over-humans-by-statistically-significant-margin/stories/201701310250, 2017. Accessed: 2017-02-07. ↗
  20. Moritz Hardt and Kunal Talwar. On the geometry of differential privacy. In ACM Symposium on Theory of Computing. ACM, 2010. find
  21. Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, 2016. find our card
  22. Carl Hempel and Paul Oppenheim. Studies in the logic of explanation. Philosophy of science, 1948. find
  23. Tin Kam Ho and Mitra Basu. Complexity measures of supervised classification problems. IEEE transactions on pattern analysis and machine intelligence, 2002. find
  24. Frank Keil. Explanation and understanding. Annu. Rev. Psychol., 2006. find
  25. Frank Keil, Leonid Rozenblit, and Candice Mills. What lies beneath? understanding the limits of understanding. Thinking and seeing: Visual metacognition in adults and children, 2004. find
  26. Been Kim, Caleb Chacha, and Julie Shah. Inferring robot task plans from human team meetings: A generative modeling approach with logic-based prior. Association for the Advancement of Artificial Intelligence, 2013. find
  27. Been Kim, Elena Glassman, Brittney Johnson, and Julie Shah. iBCM: Interactive bayesian case model empowering humans via intuitive interaction. 2015a. find
  28. Been Kim, Julie Shah, and Finale Doshi-Velez. Mind the gap: A generative approach to interpretable feature selection and extraction. In Advances in Neural Information Processing Systems, 2015b. find
  29. Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1675–1684. ACM, 2016. find
  30. Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. Research methods in human-computer interaction. John Wiley & Sons, 2010. find
  31. Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. arXiv preprint arXiv:1606.04155, 2016. find
  32. Tania Lombrozo. The structure and function of explanations. Trends in cognitive sciences, 10(10): 464–470, 2006. find
  33. Yin Lou, Rich Caruana, and Johannes Gehrke. Intelligible models for classification and regression. In ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2012. find
  34. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. find
  35. Ian Neath and Aimee Surprenant. Human Memory. 2003. find
  36. Clemens Otte. Safe and interpretable machine learning: A methodological review. In Computational Intelligence in Intelligent Data Analysis. Springer, 2013. find
  37. Parliament and Council of the European Union. General data protection regulation. 2016. find
  38. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “why should i trust you?”: Explaining the predictions of any classifier. arXiv preprint arXiv:1602.04938, 2016. find our card
  39. Salvatore Ruggieri, Dino Pedreschi, and Franco Turini. Data mining for discrimination discovery. ACM Transactions on Knowledge Discovery from Data, 2010. find
  40. Eric Schulz, Joshua Tenenbaum, David Duvenaud, Maarten Speekenbrink, and Samuel Gershman. Compositional inductive biases in function learning. bioRxiv, 2016. find
  41. D Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems, 2015. find
  42. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016. find
  43. Lior Jacob Strahilevitz. Privacy versus antidiscrimination. University of Chicago Law School Working Paper, 2008. find
  44. Adi Suissa-Peleg, Daniel Haehn, Seymour Knowles-Barley, Verena Kaynig, Thouis R Jones, Alyssa Wilson, Richard Schalek, Jeffery W Lichtman, and Hanspeter Pfister. Automatic neural reconstruction from petavoxel of electron microscopy data. Microscopy and Microanalysis, 2016. find
  45. Vincent Toubiana, Arvind Narayanan, Dan Boneh, Helen Nissenbaum, and Solon Barocas. Adnos-tic: Privacy preserving targeted advertising. 2010. find
  46. Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49–60, 2014. find
  47. Kush Varshney and Homa Alemzadeh. On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. CoRR, 2016. find
  48. Fulton Wang and Cynthia Rudin. Falling rule lists. In AISTATS, 2015. find
  49. Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille. Bayesian rule sets for interpretable classification. In International Conference on Data Mining, 2017. find
  50. Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z Gajos, Walter S Lasecki, and Neil Heffernan. Axis: Generating explanations at scale with learnersourcing and machine learning. In ACM Conference on Learning@ Scale. ACM, 2016. find
  51. Andrew Wilson, Christoph Dann, Chris Lucas, and Eric Xing. The human kernel. In Advances in Neural Information Processing Systems, 2015. find