Elamur Research⌕
☰
home / papers / 2017 / A general approach for predicting the behavior of the Supreme Court of the United States

A general approach for predicting the behavior of the Supreme Court of the United States

Paper · 2017 · Daniel Martin Katz, Michael J. Bommarito and Josh Blackman

Summary

The paper asks whether one machine-learning model can forecast US Supreme Court decisions across its whole history, not just during one stable bench. Earlier models either worked only for one fixed set of nine Justices or needed some of the votes in a case to predict the rest. The authors set three design goals: generality (the model keeps working as membership changes), consistency (it beats baselines steadily over time, issues and Justices) and out-of-sample applicability (it uses only information available before the decision). They build a random forest that is retrained as time moves forward. It uses features from the Supreme Court Database (SCDB) plus engineered features. The model predicts more than 240,000 individual Justice votes and about 28,000 case outcomes from 1816 to 2015. It reaches 70.2% accuracy on case outcomes and 71.9% on Justice votes. Over the last century it beats a strong in-sample-tuned baseline by almost 5 points. This shows that legal forecasts can be backtested and reproduced, and the authors offer the model as a reference point for later work.

Key points

  • Data source: SCDB, with up to 240 variables per case, many of them high-cardinality categoricals. For example, ISSUE has 384 values. The database comes in two releases: Modern (from 1946) and Legacy (from 1791).
  • Prediction targets: whether the Court affirms or reverses, and how each Justice votes. Justice votes are evaluated in both a 3-class and a 2-class setup.
  • Headline accuracy: 70.2% for cases and 71.9% for Justice votes over nearly two centuries. The model starts to beat the baseline consistently from roughly the post-Civil War, Fuller Court era onward.
  • Main baseline: a finite-memory null model with window M = 10.
  • Significance testing: one-sided paired t-tests and Wilcoxon rank-sum tests on per-term accuracy series, plus binomial tests on individual predictions. All show significant outperformance.
  • Reproducibility: code and data are available on GitHub (mjbommar/scotus-predict-v2).

Interesting ideas

  • I1. Strict temporal discipline: every prediction uses only information that existed before that decision, with the model updated step by step through time. This guards against leakage in any time-ordered task, such as trade flows or news classification.
  • I2. Baselines that are hard to beat: besides trivial baselines, the authors compare against an in-sample-optimized null model and a sliding-window "recent history" model. Beating a well-tuned naive predictor is a more honest claim of progress than beating a majority class.
  • I3. Consistency as a goal in its own right: they track per-term and per-Justice performance, displayed as a heatmap of wins and losses against the baseline. This rewards models that are reliably better, not just better on average. That matters for non-stationary domains where the data distribution shifts over time.
  • I4. Two kinds of significance tests: tests on aggregate per-period accuracy (parametric and non-parametric) are combined with a test on individual predictions. Together they show the gain holds both over time and case by case.
  • I5. Building in changing actors: the model is designed to keep working when participants change, through recusals, retirements or benches of different sizes. This is a useful pattern for prediction over panels with entities entering and leaving, such as companies, legislators or regions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🎓Illinois Institute of TechnologyChicago · United States18 works · 2,097 citations🎓Chicago Kent College of LawChicago · United States3 works · 920 citations🎓South Texas College of LawHouston · United States2 works · 987 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Social Sciences · 1,954 worksPolitical Science and International Relations · 206 worksLaw · 84 worksEconomics, Econometrics and Finance · 23 worksEconomics and Econometrics · 16 works

🧭 Topics 1Research subjects and languages the work is about.Legal AI · 150 works

Cite

Cite as Daniel Martin Katz, Michael J. Bommarito and Josh Blackman. 2017. A general approach for predicting the behavior of the Supreme Court of the United States. PLOS ONE.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1371/journal.pone.0174698

@inproceedings{katz2017general,
  title = {{A general approach for predicting the behavior of the Supreme Court of the United States}},
  author = {Daniel Martin Katz, Michael J. Bommarito and Josh Blackman},
  booktitle = {PLOS ONE},
  year = {2017},
  doi = {10.1371/journal.pone.0174698},
  url = {https://doi.org/10.1371/journal.pone.0174698}
}

References

41 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin AD, Quinn KM, Ruger TW, Kim PT. Competing approaches to predicting supreme court decision making. Perspectives on Politics, 2004; 2(4):761–767. https://doi.org/10.1017/S1537592704040502 ↗
  2. Guimerà R., Sales-Pardo M. Justice blocks and predictability of us supreme court votes. PloS One, 2011; 6(11):e27188. https://doi.org/10.1371/journal.pone.0027188 PMID: 22096533 ↗
  3. Ruger TW, Kim PT, Martin AD, Quinn KM. The supreme court forecasting project: Legal and political science approaches to predicting supreme court decisionmaking. Columbia Law Review, 2004; 104 (4):1150–1209. https://doi.org/10.2307/4099370 ↗
  4. Breiman L. Random forests. Machine Learning, 2001; 45(1):5–32. https://doi.org/10.1023/ A:1010933404324 ↗
  5. Katz DM. Quantitative legal prediction—or—how i learned to stop worrying and start preparing for the data driven future of the legal services industry. Emory Law Journal, 2013; 62(4): 909–966. find
  6. Breiman L, Friedman J, Stone CJ, Olshen RA. (1984) Classification and regression trees. New York: CRC press. 358 p. find
  7. Saad D. (1999) On-line learning in neural networks. Cambridge: Cambridge University Press. 398 p. ↗
  8. Shalev-Shwartz S. Online learning and online convex optimization. Foundations and Trends in Machine Learning, 2011; 4(2):107–194. https://doi.org/10.1561/2200000018 ↗
  9. Casillas CJ, Enns PK, Wohlfarth PC. How public opinion constrains the US Supreme Court. American Journal of Political Science, 2011; 55(1):74–88. https://doi.org/10.1111/j.1540-5907.2010.00485.x ↗
  10. Segal JA. Separation-of-powers games in the positive theory of congress and courts. American Political Science Review, 1997; 91(1):28–44. https://doi.org/10.2307/2952257 ↗
  11. Epstein L, Martin AD, Quinn KM, Segal JA. Ideological drift among supreme court justices: Who, when, and how important. Northwestern University Law Review, 2007; 101(4):1483–1542. find
  12. Martin AD, Quinn KM. Assessing preference change on the us supreme court. Journal of Law, Economics, and Organization, 2007; 23(2):365–385. https://doi.org/10.1093/jleo/ewm028 ↗
  13. Calderia GA, Zorn C. Of time and consensual norms in the Supreme Court. American Journal of Political Science, 1998; 42(3):874–902. https://doi.org/10.2307/2991733 ↗
  14. Ho DE, Quinn KM. Did a Switch in Time Save Nine? Journal of Legal Analysis, 2010; 2(1):69–113. https://doi.org/10.1093/jla/2.1.69 ↗
  15. Leicht EA, Clarkson G, Shedden K, Newman MEJ. Large-scale structure of time evolving citation networks. The European Physical Journal B, 2007; 59(1):75–83. https://doi.org/10.1140/epjb/e2007-00271-7 ↗
  16. Spaeth HJ, Epstein L, Martin AD, Segal JA, Ruger TJ, Benesh SC. 2016 Supreme Court Database, Version 2016 Legacy Release v01. (SCDB_Legacy_01) http://Supremecourtdatabase.org ↗
  17. Segal JA, Spaeth HJ (2002) The Supreme Court and the attitudinal model revisited. Cambridge: Cambridge University Press. 480 p. ↗
  18. Bailey MA, Maltzman F. Does legal doctrine matter? unpacking law and policy preferences on the us supreme court. American Political Science Review, 2008; 102(3):369–384. https://doi.org/10.1017/ S0003055408080283 ↗
  19. Benjamin SM, Desmarais BA. Standing the test of time: The breadth of majority coalitions and the fate of us supreme court precedents. Journal of Legal Analysis, 2012; 4(2):445–469. https://doi.org/10. 1093/jla/las016 ↗
  20. Martin AD, Quinn KM. Dynamic ideal point estimation via markov chain monte carlo for the US Supreme Court, 1953–1999. Political Analysis, 2002; 10(2):134–153. https://doi.org/10.1093/pan/10.2.134 ↗
  21. Segal JA, Epstein L, Cameron CM, Spaeth HJ. Ideological values and the votes of US Supreme Court justices revisited, The Journal of Politics, 1995; 57(3):812–823. https://doi.org/10.2307/2960194 ↗
  22. Lee ED, Broedersz CP, Bialek W. Statistical Mechanics of the US Supreme Court, Journal of Statistical Physics, 2015; 160(2): 275–301. https://doi.org/10.1007/s10955-015-1253-6 ↗
  23. Segal JA, Spaeth HJ. The influence of stare decisis on the votes of united states supreme court justices. American Journal of Political Science, 1996; 40(4):971–1003. https://doi.org/10.2307/2111738 ↗
  24. Shapiro C. Coding complexity: Bringing law to the empirical analysis of the supreme court. Hastings Law Journal, 2008; 160(2):60:477. find
  25. Katz DM, Bommarito MJ, Blackman J. Predicting the Behavior of the Supreme Court of the United States: A General Approach. 2014; arXiv:1407.6333 ↗
  26. Hofer RE. Supreme Court Reversal Rates: Evaluating the Federal Courts of Appeals. American Bar Association—Landslide, 2010; 2(3):8–11. find
  27. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011; 12:2825–2830. find
  28. Caruana R, Niculescu-Mizil A. An empirical comparison of supervised learning algorithms. In: Proceedings of the 23rd international conference on Machine learning. ACM; 2006 p. 161–168. find
  29. Chollet F. Keras: Deep Learning library for TensorFlow and Theano, Github, 2015; https://github.com/ fchollet/keras ↗
  30. Louppe G. Understanding Random Forests: From Theory to Practice Dissertation, University of Liege, Belgium, 2014. find
  31. Katz DM, Bommarito MJ, Soellinger T, Chen JM. Law on the Market? Evaluating the Securities Market Impact of Supreme Court Decisions 2015; arXiv:1508.05751 ↗
  32. Aletras N, Tsarapatsanis D, Preoţiuc-Pietro D, Lampos V. Predicting judicial decisions of the European Court of Human Rights: A natural language processing perspective. PeerJ Computer Science, 2016; 2: e93. https://doi.org/10.7717/peerj-cs.93 ↗
  33. Harbert T. The Law Machine. IEEE Spectrum, 2013; 50(11):31–54. https://doi.org/10.1109/MSPEC. 2013.6655836 ↗
  34. Ashley KD, Brüninghaus S. Automatically Classifying Case Texts and Predicting Outcomes. Artificial Intelligence and Law. 2009; 17(2):125–65. https://doi.org/10.1007/s10506-009-9077-9 ↗
  35. McShane BB, Watson OP, Baker T, Griffith SJ. Predicting securities fraud settlements and amounts: a hierarchical Bayesian model of federal securities class action lawsuits. Journal of Empirical Legal Studies. 2012; 9(3):482–510. https://doi.org/10.1111/j.1740-1461.2012.01260.x ↗
  36. Cormack GV, Grossman MR. Evaluation of machine-learning protocols for technology-assisted review in electronic discovery. In: Proceedings of the 37th International ACM SIGIR conference on Research & development in information retrieval ACM; 2014 p. 153–162. find
  37. Surden H. Machine learning and law. Washington Law Review. 2014; 89(1):87–115. find
  38. Sanga S. Choice of law: an empirical analysis. Journal of Empirical Legal Studies. 2014; 11(4): 894– 928. https://doi.org/10.1111/jels.12059 ↗
  39. Talley E, O’Kane D. The measure of a MAC: A machine-learning protocol for analyzing force majeure clauses in M&A agreements. Journal of Institutional and Theoretical Economics. 2012; 168(1): 181– 201. https://doi.org/10.1628/093245612799440177 ↗
  40. Blackman J, Aft A, Carpenter C. Fantasyscotus: Crowdsourcing a prediction market for the supreme court. Northwestern Journal of Technology and Intellectual Property, 2012; 10(3): 125–168. find
  41. Katz DM, Bommarito MJ. The Three Forms of (Legal) Prediction—Experts, Crowds and Algorithms. Lecture presented at the Vanderbilt Conference on Artificial Intelligence and Law, 2016; Vanderbilt Law School. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/katz-2017-general-approach-predicting-behavior/