Elamur Research
☰
home / papers / 2018 / Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1811.10154

Summary

This perspective piece argues that in consequential areas such as criminal justice, healthcare and lending, practitioners should build models that are transparent by design rather than add post-hoc explanation tools to opaque predictors. Rudin separates "explainable" black boxes from inherently interpretable models. The latter have a constrained form, using sparsity, monotonicity, additivity or case-based reasoning, so the model serves as its own explanation. She disputes the common belief that accuracy must be traded for interpretability. On structured data with meaningful features, simple and complex learners usually perform about equally, and iterative data refinement guided by interpretation often matters more than the choice of algorithm. She also shows that post-hoc explanations are inevitably imperfect copies of the original model and can mislead users. The paper describes technical challenges for interpretable ML, gives application examples, and offers the Rashomon set argument for why accurate simple models are often available. Its importance lies in pushing researchers and regulators to require genuine interpretability before accepting black boxes in high-stakes settings.

Key points

  • New York City power-grid failure prediction: The data were messy and included trouble tickets and cable records dating back to the 1890s. Algorithms run on a fixed dataset differed by at most 1%. Real gains came from interpreting results and reprocessing the data, and the strongest predictors were sparse models.
  • 2018 FICO Explainable ML Challenge: Entrants were asked to build a black box and explain it afterwards. A globally interpretable additive model won the FICO Recognition Prize and performed no worse than black-box entries.
  • Fidelity problem: An explanation that matches its model 90% of the time is still wrong in one case out of ten. In a recidivism example, a mimic model could attribute predictions to race, even though the real model relies on age and criminal history.
  • DARPA XAI trade-off chart: The paper criticizes this widely shown chart as fictional, with unquantified axes and no underlying data.
  • Rashomon set argument: Finite data typically admit many near-optimal models, for instance those within a small margin of boosted decision trees. A large set of this kind likely includes an interpretable member.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

154 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 148 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.10154 · arXiv 1811.10154

@misc{rudin2018stop,
  title = {{Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead}},
  author = {Rudin, Cynthia},
  year = {2018},
  doi = {10.48550/arxiv.1811.10154},
  eprint = {1811.10154},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.10154}
}

References

67 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wexler R. When a Computer Program Keeps You in Jail: How Computers are Harming Criminal Justice. New York Times. 2017 June 13;. find
  2. McGough M. How bad is Sacramento’s air, exactly? Google results appear at odds with reality, some say. Sacramento Bee. 2018 August 7;. find
  3. Varshney KR, Alemzadeh H. On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. Big Data. 2016 10;5. find
  4. Freitas AA. Comprehensible classification models: a position paper. ACM SIGKDD Explorations Newsletter. 2014 Mar;15(1):1–10. find
  5. Kodratoff Y. The comprehensibility manifesto. KDD Nugget Newsletter. 1994;94(9). find
  6. Huysmans J, Dejaeger K, Mues C, Vanthienen J, Baesens B. An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decision Support Systems. 2011;51(1):141–154. find
  7. Rüping S. Learning Interpretable Models. Universität Dortmund; 2006. find
  8. Gupta M, Cotter A, Pfeifer J, Voevodski K, Canini K, Mangylov A, et al. Monotonic calibrated interpolated look-up tables. Journal of Machine Learning Research. 2016;17(109):1–47. find
  9. Lou Y, Caruana R, Gehrke J, Hooker G. Accurate Intelligible Models with Pairwise Interactions. In: Proceedings of 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM; 2013. . find
  10. Miller G. The magical number seven, plus or minus two: Some limits on our capacity for processing information. The Psychological Review. 1956;63:81–97. find
  11. Cowan N. The magical mystery four how is working memory capacity limited, and why? Current directions in psychological science. 2010;19(1):51–57. find
  12. Wang J, Oh J, Wang H, Wiens J. Learning Credible Models. In: Proceedings of 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM; 2018. p. 2417–2426. find
  13. Rudin C. Please Stop Explaining Black Box Models for High Stakes Decisions. In: Proceedings of NeurIPS 2018 Workshop on Critiquing and Correcting Trends in Machine Learning; 2018. . find
  14. Holte RC. Very simple classification rules perform well on most commonly used datasets. Machine Learning. 1993;11(1):63–91. find
  15. Fayyad U, Piatetsky-Shapiro G, Smyth P. From data mining to knowledge discovery in databases. AI Magazine. 1996;17:37–54. find
  16. Chapman P, et al. CRISP-DM 1.0 - Step-by-step data mining guide. SPSS; 2000. find
  17. Agrawal D, Bernstein P, Bertino E, Davidson S, Dayal U, Franklin M, et al. Challenges and Opportunities with Big Data: A white paper prepared for the Computing Community Consortium committee of the Computing Research Association; 2012. Available from: http://cra.org/ccc/resources/ccc-led-whitepapers… ↗
  18. Defense Advanced Research Projects Agency. Broad Agency Announcement, Explainable Artificial Intelligence (XAI), DARPA-BAA-16-53; 2016. Published August 10. Available from https://www.darpa.mil/ attachments/DARPA-BAA-16-53.pdf. ↗
  19. Hand D. Classifier Technology and the Illusion of Progress. Statist Sci. 2006;21(1):1–14. find
  20. Rudin C, Passonneau R, Radeva A, Dutta H, Ierome S, Isaac D. A Process for Predicting Manhole Events In Manhattan. Machine Learning. 2010;80:1–31. find
  21. Rudin C, Ustun B. Optimized Scoring Systems: Toward Trust in Machine Learning for Healthcare and Criminal Justice. Interfaces. 2018;48:399–486. Special Issue: 2017 Daniel H. Wagner Prize for Excellence in Operations Research Practice September-October 2018. find
  22. Chen C, Lin K, Rudin C, Shaposhnik Y, Wang S, Wang T. An Interpretable Model with Globally Consistent Explanations for Credit Risk. In: Proceedings of NeurIPS 2018 Workshop on Challenges and Opportunities for AI in Financial Services: the Impact of Fairness, Explainability, Accuracy, and Privacy; 20… find
  23. Mittelstadt B, Russell C, Wachter S. Explaining Explanations in AI. In: In Proceedings of Fairness, Accountability, and Transparency (FAT*); 2019. . find
  24. Flores AW, Lowenkamp CT, Bechtel K. False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country to Predict Future Criminals”. Federal probation. 2016 September;80(2):38–46. find
  25. Angwin J, Larson J, Mattu S, Kirchner L. Machine Bias. ProPublica; 2016. Available from: https://www. propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. find
  26. Larson J, Mattu S, Kirchner L, Angwin J. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica; 2016. https://www.propublica.org/article/ how-we-analyzed-the-compas-recidivism-algorithm. ↗
  27. Rudin C, Wang C, Coker B. The age of secrecy and unfairness in recidivism prediction. arXiv e-prints 1811 00731 [applied statistics]. 2018 Nov;. find
  28. Checkermallow. Canis lupus winstonii (Siberian Husky); 2016. Public domain image. https://www.flickr.com/photos/132792051@N06/28302196071/in/ photolist-K7Y9RM-utZTV9-QWJmHo-QAEdSE-QAE3pL-TvjNJu-tziyrj-EWFwEx-DWb7T4-DTRAWu-CYLBpP- ↗
  29. Brennan T, Dieterich W, Ehret B. Evaluating the Predictive Validity of the COMPAS Risk and Needs Assessment System. Criminal Justice and Behavior. 2009 January;36(1):21–40. find
  30. Zeng J, Ustun B, Rudin C. Interpretable classification models for recidivism prediction. Journal of the Royal Statistical Society: Series A (Statistics in Society). 2017;180(3):689–722. find
  31. Tollenaar N, van der Heijden PGM. Which method predicts recidivism best?: a comparison of statistical, machine learning and data mining predictive models. Journal of the Royal Statistical Society: Series A (Statistics in Society). 2013;176(2):565–584. find
  32. Angelino E, Larus-Stone N, Alabi D, Seltzer M, Rudin C. Certifiably optimal rule lists for categorical data. Journal of Machine Learning Research. 2018;19:1–79. find
  33. Mannshardt E, Naess L. Air quality in the USA. Significance. 2018 Oct;15:24–27. find
  34. Zech JR, et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLoS Med. 2018;15(e1002683). find
  35. Chang A, Rudin C, Cavaretta M, Thomas R, Chou G. How to Reverse-Engineer Quality Rankings. Machine Learning. 2012 September;88:369–398. find
  36. Goodman B, Flaxman S. EU regulations on algorithmic decision-making and a ‘right to explanation’. AI Magazine. 2017;38(3). find
  37. Wachter S, Mittelstadt B, Russell C. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology. 2018;1(2). find
  38. Quinlan JR. C4. 5: programs for machine learning. vol. 1. Morgan Kaufmann; 1993. find our card
  39. Breiman L, Friedman J, Stone CJ, Olshen RA. Classification and regression trees. CRC press; 1984. find
  40. Auer P, Holte RC, Maass W. Theory and Applications of Agnostic PAC-Learning with Small Decision Trees. In: Machine Learning Proceedings 1995. San Francisco (CA): Morgan Kaufmann; 1995. p. 21 – 29. find
  41. Wang F, Rudin C. Falling Rule Lists. In: Proceedings of Machine Learning Research Vol. 38: Artificial Intelligence and Statistics (AISTATS); 2015. p. 1013–1022. find
  42. Chen C, Rudin C. An optimization approach to learning falling rule lists. In: Proceedings of Machine Learning Research Vol. 84: Artificial Intelligence and Statistics (AISTATS); 2018. p. 604–612. find
  43. Burgess EW. Factors determining success or failure on parole; 1928. Illinois Committee on Indeterminate-Sentence Law and Parole Springfield, IL. find
  44. Ustun B, Rudin C. Optimized Risk Scores. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD); 2017. . find
  45. Ustun B, Rudin C. Supersparse linear integer models for optimized medical scoring systems. Machine Learning. 2015;p. 1–43. find
  46. Carrizosa E, Martín-Barragán B, Morales DR. Binarized support vector machines. INFORMS Journal on Computing. 2010;22(1):154–167. find
  47. Sokolovska N, Chevaleyre Y, Zucker JD. A Provable Algorithm for Learning Interpretable Scoring Systems. In: Proceedings of Machine Learning Research Vol. 84: Artificial Intelligence and Statistics (AISTATS); 2018. p. 566–574. find
  48. Ustun B, et al. The World Health Organization Adult Attention-Deficit/Hyperactivity Disorder Self-Report Screening Scale for DSM-5. JAMA Psychiatry. 2017;74(5):520–526. find
  49. Chen C, Li O, Tao C, Barnett A, Su J, Rudin C. This Looks Like that: Deep Learning for Interpretable Image Recognition. In: Neural Information Processing Systems (NeurIPS); 2019. . find
  50. O’Malley D. Clay-colored Sparrow; 2014. Public domain image. https://www.flickr.com/photos/ 62798180@N03/11895857625/. ↗
  51. ksblack99. Clay-colored Sparrow; 2018. Public domain image. https://www.flickr.com/photos/ ksblack99/42047311831/. ↗
  52. Schmierer A. Clay-colored Sparrow; 2017. Public domain image. https://flic.kr/p/T6QVkY. ↗
  53. Schmierer A. Clay-colored Sparrow; 2015. Public domain image. https://flic.kr/p/rguC7K. ↗
  54. Schmierer A. Clay-colored Sparrow; 2015. Public domain image. https://www.flickr.com/photos/ sloalan/16585472235/. ↗
  55. Li O, Liu H, Chen C, Rudin C. Deep Learning for Case-based Reasoning through Prototypes: A Neural Network that Explains its Predictions. In: Proceedings of AAAI Conference on Artificial Intelligence (AAAI); 2018. p. 3530–3537. find
  56. Gallagher N, et al. Cross-Spectral Factor Analysis. In: Proceedings of Advances in Neural Information Processing Systems 30 (NeurIPS). Curran Associates, Inc.; 2017. p. 6842–6852. find
  57. Wang F, Rudin C, Mccormick TH, Gore JL. Modeling recovery curves with application to prostatectomy. Biostatistics. 2018;p. kxy002. Available from: http://dx.doi.org/10.1093/biostatistics/kxy002. — [doi:10.1093/biostatistics/kxy002](https://doi.org/10.1093/biostatistics/kxy002) ↗
  58. Lou Y, Caruana R, Gehrke J. Intelligible Models for Classification and Regression. In: Proceedings of Knowledge Discovery in Databases (KDD). ACM; 2012. . find
  59. Semenova L, Parr R, Rudin C. A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning; 2018. In progress. find
  60. Razavian N, et al. Population-Level Prediction of Type 2 Diabetes From Claims Data and Analysis of Risk Factors. Big Data. 2015;3(4). find
  61. Ustun B, Spangher A, Liu Y. Actionable Recourse in Linear Classification. In: ACM Conference on Fairness, Accountability and Transparency (FAT*); 2019. . find
  62. Su G, Wei D, Varshney KR, Malioutov DM. Interpretable Two-Level Boolean Rule Learning for Classification. In: Proceedings of ICML Workshop on Human Interpretability in Machine Learning; 2016. p. 66–70. find
  63. Dash S, Günlük O, Wei D. Boolean Decision Rules via Column Generation. In: 32nd Conference on Neural Information Processing Systems (NeurIPS); 2018. . find
  64. Wang T, Rudin C, Doshi-Velez F, Liu Y, Klampfl E, MacNeille P. A Bayesian Framework for Learning Rule Sets for Interpretable Classification. Journal of Machine Learning Research. 2017;18(70):1–37. find
  65. Rijnbeek PR, Kors JA. Finding a Short and Accurate Decision Rule in Disjunctive Normal Form by Exhaustive Search. Machine Learning. 2010 Jul;80(1):33–62. find
  66. Goh ST, Rudin C. Box Drawings for Learning with Imbalanced Data. In: Proceedings of the 20th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD); 2014. . find
  67. Murdoch WJ, Singh C, Kumbier K, Abbasi-Asl R, Yu B. Interpretable machine learning: definitions, methods, and applications. arXiv e-prints: 1901 04592 [statistical machine learning]. 2019 Jan;. find