Elamur Research
☰
home / papers / 2023 / Can GPT-3 Perform Statutory Reasoning?

Can GPT-3 Perform Statutory Reasoning?

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2302.06100

Summary

The paper asks whether a large language model can apply written legal rules to stated facts, a core skill for lawyers. The authors test OpenAI's text-davinci-003 (GPT-3) on SARA, a benchmark built from nine sections of the U.S. tax code. They compare zero-shot prompting, 4-shot prompting with dynamically chosen similar training cases, and 10-shot chain-of-thought prompting, all at temperature 0. GPT-3 beats earlier published systems on SARA, reaching about 71% accuracy, but it still makes obvious mistakes.

To explain these errors, the authors run two probes. First, they check what the model already knows about the real U.S. Code and find that this knowledge is only partial. Second, they generate artificial statutes made of invented words, so the model cannot have seen them during training. Even on very shallow rule hierarchies, GPT-3 often misjudges which subsection applies to a given fact.

The results suggest that statutory reasoning is still an open challenge for LLMs and a useful benchmark for future models.

Key points

  • SARA setup: The dataset has 376 hand-built cases, and the 100 numeric tax-liability cases were dropped. The remaining Entailment/Contradiction cases were evaluated on 72 test items that involve numbers and 28 that do not. A Prolog version of SARA solves every case correctly.
  • SARA result: The best GPT-3 accuracy was about 71%, which exceeds the prior state of the art.
  • Synthetic statutes: The authors varied width and depth and used either nonce terms (e.g. "portle") or ids (e.g. "m77"). Zero-shot accuracy on the simplest 2-wide, 2-deep statutes was only about 78%.
  • Statute vs. sentence form: Accuracy was nearly the same whether the rules were written as statute sections or as plain sentences (6537 vs. 6562 correct out of 9000). Most errors were false positives, i.e. 2204 of 2272 errors in the nonce runs.
  • Two-shot prompting: Adding two worked examples raised accuracy to 100% on 2-wide, 2-deep statutes. It fell to 81% on 3-wide, 3-deep ones. The code is available at github.com/BlairStanek/gpt-statutes, and the paper appeared at ICAIL 2023.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning · Law

Cited by

73 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 68 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.06100 · arXiv 2302.06100

@misc{blairstanek2023can,
  title = {{Can GPT-3 Perform Statutory Reasoning?}},
  author = {Blair-Stanek, Andrew and Holzenberger, Nils and Durme, Benjamin Van},
  year = {2023},
  doi = {10.48550/arxiv.2302.06100},
  eprint = {2302.06100},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.06100}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sheng Bi, Zhiyao Zhou, Lu Pan, and Guilin Qi. 2022. Judicial knowledge-enhanced magnitude-aware reasoning for numerical legal judgment prediction. Artificial Intelligence and Law (2022), 1–34. https://doi.org/10.1007/s10506-022-09337-4 ↗
  2. Michael Bommarito, II and Daniel Martin Katz. 2022. GPT Takes the Bar Exam. CoRR abs/2212.14402 (2022). https://doi.org/10.48550/arXiv.2212.14402 arXiv:2212.14402 ↗
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/ hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html ↗
  4. Jonathan H. Choi, Kristin E. Hickman, Amy Monahan, and Daniel B. Schwarcz. 2023. ChatGPT Goes to Law School. SSRN abs/2212.14402 (2023). https://doi.org/10.2139/ssrn.4335905 SSRN:2212.14402 ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423 ↗
  6. Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 4497–4512. https://aclanthology.org/2022.findings-emnlp.329 ↗
  7. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=d7KBjmI3GmQ ↗ our card
  8. Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020. A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering. In Proceedings of the Natural Legal Language Processing Workshop 2020 co-located with the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2020), Virtual Workshop, August 24, 2020 (CEUR Workshop Proceedings, Vol. 2645), Nikolaos Aletras, Ion Androutsopoulos, Leslie Barrett, Adam Meyers, and Daniel Preotiuc-Pietro (Eds.). CEUR-WS.org, 31–38. http://ceur-ws.org/Vol-2645/paper5.pdf ↗
  9. Nils Holzenberger and Benjamin Van Durme. 2021. Factoring Statutory Reasoning as Language Understanding Challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, 2742–2758. https://doi.org/10.18653/v1/2021.acl-long.213 ↗
  10. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed Prompting: A Modular Approach for Solving Complex Tasks. CoRR abs/2210.02406 (2022). https://doi.org/10.48550/arXiv.2210.02406 arXiv:2210.02406 ↗
  11. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. CoRR abs/2205.11916 (2022). https://doi.org/10.48550/arXiv.2205.11916 arXiv:2205.11916 ↗ our card
  12. Jason T. Lam, David Liang, Samuel Dahan, and Farhana H. Zulkernine. 2020. The Gap between Deep Learning and Law: Predicting Employment Notice. In Proceedings of the Natural Legal Language Processing Workshop 2020 co-located with the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2020), Virtual Workshop, August 24, 2020 (CEUR Workshop Proceedings, Vol. 2645), Nikolaos Aletras, Ion Androutsopoulos, Leslie Barrett, Adam Meyers, and Daniel Preotiuc-Pietro (Eds.). CEUR-WS.org, 52–56. http://ceur-ws.org/Vol-2645/short2.pdf ↗
  13. Sarah B Lawsky. 2017. A logic for statutes. Fla. Tax Rev. 21 (2017), 60. find
  14. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. CoRR abs/2209.09513 (2022). https://doi.org/10.48550/arXiv.2209.09513 arXiv:2209.09513 ↗
  15. Matt Post. 2018. A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers. Association for Computational Linguistics, Belgium, Brussels, 186–191. https://www.aclweb.org/anthology/W18-6319 ↗ our card
  16. Juliano Rabelo, Randy Goebel, Mi-Young Kim, Yoshinobu Kano, Masaharu Yoshioka, and Ken Satoh. 2022. Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021. Rev. Socionetwork Strateg. 16, 1 (2022), 111–133. https://doi.org/10.1007/s12626-022-00105-z ↗
  17. Abulhair Saparov and He He. 2022. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. CoRR abs/2210.01240 (2022). https://doi.org/10.48550/arXiv.2210.01240 arXiv:2210.01240 ↗ our card
  18. Jaromír Šavelka and Kevin D Ashley. 2021. Discovering Explanatory Sentences in Legal Case Decisions Using Pre-trained Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2021. 4273–4283. find
  19. Jaromír Šavelka and Kevin D Ashley. 2021. Legal information retrieval for understanding statutory terms. Artificial Intelligence and Law (2021), 1–45. find
  20. Luke Thorburn and Ariel Kruger. 2022. Optimizing Language Models for Argumentative Reasoning. In Proceedings of the 1st Workshop on Argumentation & Machine Learning co-located with 9th International Conference on Computational Models of Argument (COMMA 2022), Cardiff, Wales, September 13th, 2022 (CEUR Workshop Proceedings, Vol. 3208), Isabelle Kuhlmann, Jack Mumford, and Stefan Sarkadi (Eds.). CEUR-WS.org, 27–44. http://ceur-ws.org/Vol-3208/paper3.pdf ↗
  21. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax. ↗
  22. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. CoRR abs/2201.11903 (2022). arXiv:2201.11903 https://arxiv.org/abs/2201.11903 ↗ our card
  23. Fangyi Yu, Lee Quartey, and Frank Schilder. 2022. Legal Prompting: Teaching a Language Model to Think Like a Lawyer. CoRR abs/2212.01326 (2022). https://doi.org/10.48550/arXiv.2212.01326 arXiv:2212.01326 ↗
  24. Eric Zelikman, Yuhuai Wu, and Noah D. Goodman. 2022. STaR: Bootstrapping Reasoning With Reasoning. CoRR abs/2203.14465 (2022). https://doi.org/10. 48550/arXiv.2203.14465 arXiv:2203.14465 ↗ our card
  25. Hanlin Zhang, Ziyang Li, Jiani Huang, Mayur Naik, and Eric Xing. 2022. Improved logical reasoning of language models via differentiable symbolic programming. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022. find
  26. Haoxi Zhong, Yuzhong Wang, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, 1250–1257. https://ojs.aaai.org/index.php/AAAI/article/view/ 5479 ↗
  27. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed H. Chi. 2022. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. CoRR abs/2205.10625 (2022). https://doi.org/10.48550/arXiv.2205.10625 arXiv:2205.10625 ↗ our card