📄 Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence

Summary

The paper asks how well large language models can apply law, using U.S. federal tax law as a testbed. Tax suits this purpose because its authority sits mainly in two fixed sources: the U.S. Code's Title 26 and the Treasury Regulations contained in the CFR. Many tax questions also have one verifiable answer that requires arithmetic as well as reading. The authors wrote Python generators that create new multiple-choice problems with random facts, names and figures, so no test item could have leaked into a training corpus. They tested successive OpenAI models, up to GPT-4. Each model ran alone, with legal text supplied through several retrieval approaches, and with few-shot examples. GPT-4 also served as the automatic grader. Accuracy improved with every newer model release. GPT-4 gained noticeably from few-shot examples and from receiving the correct source passages, but it still did not reach the level of an expert tax lawyer. The study offers a contamination-resistant way to track emerging legal reasoning in LLMs.

Key points

• Two synthetic exams. The CFR exam has three question types and the U.S. Code exam has four, with 100 questions per type.
• What each generated item contains. Every item includes answer options, the correct choice, an explanation, and a citation to the governing provision.
• Topics covered. Questions span "check the box" entity classification, restricted and unrestricted property as compensation, part gift/part sale, basis of gifted or inherited property, and the Section 199A qualified business income deduction.
• Automated grading. More than 28,000 answers across all experiments were scored automatically by GPT-4, which compared each model's output against the ground truth.
• Best-performing setup. The strongest configuration (the "mega_run" prompt) gave GPT-4 the relevant ("gold truth") legal sources, three solved examples, and a step-by-step reasoning cue.