Elamur Research⌕
☰
home / papers / 2020 / A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering

A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering

Paper · 2020 · Nils Holzenberger, Andrew Blair-Stanek and Benjamin Van Durme

Summary

The paper frames statutory reasoning as a distinct NLP challenge: a system must apply legal rules, each stated only once in natural language, to a factual scenario that is also written in prose. Much machine reading instead relies on knowledge absorbed from large-scale text statistics. To study this problem, the authors release SARA (StAtutory Reasoning Assessment), built from a simplified excerpt of the US Internal Revenue Code. It contains hand-written cases framed as entailment, deciding whether a given subsection applies, and a second set of cases where the answer is the exact tax amount owed. They also provide a legal-domain text corpus and a hand-built Prolog encoding of the statutes. The Prolog system solves every example. Standard machine reading models perform poorly, even after adaptation to legal text. The authors argue that legal rules hold simply because they were written and agreed upon, which makes statutes an unusually clean test of whether models can follow prescriptive instructions. Because expert annotation is expensive, the results raise the question of whether research should focus on better semantic parsing or on more data-efficient machine reading.

Key points

  • The statutes come from IRC sections §1, §2, §63, §68, §151, §152, §3301, §3306 and §7703, covering individual tax, marital status, dependents, deductions and employment.
    • They were pruned to stay manageable and to reference only each other.
    • The original section numbering was kept.
  • The entailment set has 276 cases, split into 176 train and 100 test. Each subsection has one case where it applies and one where it does not. A law professor checked the cases for plausibility.
  • The tax-amount set has 100 cases, split 80/20. Each answer is an integer that requires the whole statute set and basic arithmetic.
  • The statutes were quantified along several dimensions:
    • 62 sections involve time.
    • 50 involve exceptions or substitutions.
    • 55 involve numerical reasoning.
    • The section hierarchy forms a tree with 193 nodes, 132 of them leaves.
    • Four common-sense notions are deliberately left undefined, including kinship and marriage ending at death.
  • The Prolog solver reaches 100% accuracy, while off-the-shelf and legal-domain-adapted machine reading baselines score low.

Interesting ideas

  • I1. Contrastive case pairs. Every rule gets one example where it applies and one where it does not, which discourages shortcut learning and isolates understanding of each rule. The same design could probe how models handle any regulatory or procedural text.
  • I2. Leakage-aware data splits. Each positive/negative pair stays in the same split, and every section is divided in the same train/test proportion. This is a cheap way to avoid leakage and imbalance in small expert-built benchmarks.
  • I3. Symbolic reference solution. A hand-coded Prolog program that solves the whole dataset confirms the task is well-defined and fully answerable. It also gives a clear upper bound to compare learned models against.
  • I4. Composed harder tasks. The tax-amount cases were created by combining simpler cases and manually fixing contradictions. Reusing annotated atomic examples this way yields multi-rule reasoning items at low annotation cost.
  • I5. Explicit difficulty profile. The authors count reasoning phenomena (time, exceptions, arithmetic, cross-references, hierarchy depth) and list the common sense left undefined. This documents what a benchmark actually tests, which is valuable when building similar resources for other legal systems or languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States248 works · 106,397 citations🎓University of Maryland, BaltimoreBaltimore · United States4 works · 457 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPolitical Science and International Relations · 206 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,685 worksQuestion answering · 487 worksLegal AI · 150 works

Cite

Cite as Nils Holzenberger, Andrew Blair-Stanek and Benjamin Van Durme. 2020. A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering. arXiv:2005.05257.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.05257 · arXiv 2005.05257

@misc{holzenberger2020dataset,
  title = {{A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering}},
  author = {Nils Holzenberger, Andrew Blair-Stanek and Benjamin {Van Durme}},
  year = {2020},
  doi = {10.48550/arxiv.2005.05257},
  eprint = {2005.05257},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.05257}
}

References

66 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2019. Caselaw Access Project. http://case.law ↗
  2. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2016. A simple but tough-to-beat baseline for sentence embeddings. (2016). find
  3. Kevin D Ashley and Stefanie Brüninghaus. 2009. Automatically classifying case texts and predicting outcomes. Artificial Intelligence and Law 17, 2 (2009), 125–165. ↗
  4. Trevor JM Bench-Capon, Gwen O Robinson, Tom W Routen, and Marek J Sergot. 1987. Logic programming for large scale applications in law: A formalisation of supplementary benefit legislation. In Proceedings of the 1st international conference on Artificial intelligence and law. 190–198. find
  5. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The Fifth PASCAL Recognizing Textual Entailment Challenge.. In TAC. find
  6. Carlo Biagioli, Enrico Francesconi, Andrea Passerini, Simonetta Montemagni, and Claudia Soria. 2005. Automatic semantics extraction in law documents. In Proceedings of the 10th international conference on Artificial intelligence and law. 133–140. find
  7. SRK Branavan, David Silver, and Regina Barzilay. 2012. Learning to win by reading manuals in a monte-carlo framework. Journal of Artificial Intelligence Research 43 (2012), 661–704. ↗
  8. Stefanie Bruninghaus and Kevin D Ashley. 2003. Predicting outcomes of case based legal arguments. In Proceedings of the 9th international conference on Artificial intelligence and law. 233–242. find
  9. Hector Neri Castañeda. 1967. Comment on D. Davidson’s “The logical forms of action sentences”. The Logic of Decision and Action (1967). find
  10. Ilias Chalkidis and Ion Androutsopoulos. 2017. A Deep Learning Approach to Contract Element Extraction.. In JURIX. 155–164. ↗
  11. Ilias Chalkidis, Ion Androutsopoulos, and Achilleas Michos. 2018. Obligation and prohibition extraction using hierarchical rnns. arXiv preprint arXiv:1805.03871 (2018). ↗
  12. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2019. Large-Scale Multi-Label Text Classification on EU Legislation. arXiv preprint arXiv:1906.02192 (2019). ↗
  13. Debajyoti Paul Chowdhury, Arghya Biswas, Tomasz Sosnowski, and Kristina Yordanova. 2020. Towards Evaluating Plan Generation Approaches with Instructional Texts. arXiv preprint arXiv:2001.04186 (2020). ↗
  14. Peter Clark, Oren Etzioni, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, Sumithra Bhakthavatsalam, et al. 2019. From’F’to’A’on the NY Regents Science Exams: An Overview of the Aristo Project. arXiv preprint arXiv:1909.01958 (2019). ↗
  15. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. Technical Report. Technical Report LRE 62-051 D-16, The FraCaS Consortium. find
  16. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop. Springer, 177–190. find
  17. Donald Davidson. 1967. The logical forms of action sentences. The Logic of Decision and Action (1967). find
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). ↗
  19. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161 (2019). ↗
  20. Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open information extraction from the web. Commun. ACM 51, 12 (2008), 68–74. ↗
  21. Edward A Feigenbaum. 1992. Expert systems: principles and practice. (1992). find
  22. David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al. 2010. Building Watson: An overview of the DeepQA project. AI magazine 31, 3 (2010), 59–79. ↗
  23. Noah S Friedland, Paul G Allen, Gavin Matthews, Michael Witbrock, David Baxter, Jon Curtis, Blake Shepard, Pierluigi Miraglia, Jurgen Angele, Steffen Staab, et al. 2004. Project halo: Towards a digital aristotle. AI magazine 25, 4 (2004), 29–29. ↗
  24. Wachara Fungwacharakorn and Ken Satoh. 2018. Legal Debugging in Propositional Legal Representation. In JSAI International Symposium on Artificial Intelligence. Springer, 146–159. find
  25. Bryan A Gardner. 2019. Black’s Law Dictionary (11 ed.). find
  26. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing. Association for Computational Linguistics, 1–9.
  27. David Gunning, Vinay K Chaudhri, Peter E Clark, Ken Barker, Shaw-Yi Chaw, Mark Greaves, Benjamin Grosof, Alice Leung, David D McDonald, Sunil Mishra, et al. 2010. Project Halo UpdateâĂŤProgress Toward Digital Aristotle. AI Magazine 31, 3 (2010), 33–58. find
  28. R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. find
  29. Herbert Lionel Adolphus Hart and Herbert Lionel Adolphus Hart. 2012. The concept of law. Oxford university press. ↗
  30. Robert Hellawell. 1980. A computer program for legal planning and analysis: Taxation of stock redemptions. Columbia Law Review 80, 7 (1980), 1363–1398. ↗
  31. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015). ↗
  32. Imran Khan, Muhammad Sher, Javed I Khan, Syed M Saqlain, Anwar Ghani, Husnain A Naqvi, and Muhammad Usman Ashraf. 2016. Conversion of legal text to a logical rules set from medical law using the medical relational model and the world rule model for a medical decision support system. In Informatics, Vol. 3. Multidisciplinary Digital Publishing Institute, 2. ↗
  33. Mi-Young Kim, Juliano Rabelo, and Randy Goebel. 2019. Statute Law Information Retrieval and Entailment. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law. 283–289. find
  34. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). ↗
  35. Anastassia Kornilova and Vladimir Eidelman. 2019. BillSum: A Corpus for Automatic Summarization of US Legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. Association for Computational Linguistics, Hong Kong, China, 48–56. https://doi.org/10.18653/v1/D19-5406 ↗
  36. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. (2001). find
  37. Robert S Ledley and Lee B Lusted. 1959. Reasoning foundations of medical diagnosis. Science 130, 3366 (1959), 9–21. ↗
  38. L Thorne McCarty. 1976. Reflections on TAXMAN: An experiment in artificial intelligence and legal reasoning. Harv. L. Rev. 90 (1976), 837. ↗
  39. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111–3119. ↗
  40. Randolph A Miller, Harry E Pople Jr, and Jack D Myers. 1982. Internist-I, an experimental computer-based diagnostic consultant for general internal medicine. New England Journal of Medicine 307, 8 (1982), 468–476. ↗
  41. Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhanava Dalvi, Matt Gardner, Bryan Kisiel, et al. 2018. Never-ending learning. Commun. ACM 61, 5 (2018), 103–115. ↗
  42. Terence Parsons. 1990. Events in the Semantics of English. Vol. 334. MIT press Cambridge, MA. find
  43. Walter G Popp and Bernhard Schlink. 1974. Judith, a computer program to advise lawyers in reasoning a case. Jurimetrics J. 15 (1974), 303. find
  44. Juliano Rabelo, Mi-Young Kim, and Randy Goebel. 2019. Combining Similarity and Transformer Methods for Case Law Entailment. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law. 290–296. find
  45. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). ↗
  46. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.
  47. Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, and Norman Sadeh. 2019. Question Answering for Privacy Policies: Combining Computational and Legal Perspectives. arXiv preprint arXiv:1911.00841 (2019). ↗
  48. Edwina L Rissland, Kevin D Ashley, and Ronald Prescott Loui. 2003. AI and Law: A fruitful synergy. Artificial Intelligence 150, 1-2 (2003), 1–15. find
  49. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. find
  50. Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. arXiv preprint arXiv:1809.01494 (2018). ↗
  51. Ken Satoh, Kento Asai, Takamune Kogawa, Masahiro Kubota, Megumi Nakamura, Yoshiaki Nishigai, Kei Shirakawa, and Chiaki Takano. 2010. PROLEG: an implementation of the presupposed ultimate fact theory of Japanese civil code by PROLOG technology. In JSAI International Symposium on Artificial Intelligence. Springer, 153–164. find
  52. Marek J. Sergot, Fariba Sadri, Robert A. Kowalski, Frank Kriwaczek, Peter Hammond, and H Terese Cory. 1986. The British Nationality Act as a logic program. Commun. ACM 29, 5 (1986), 370–386. ↗
  53. David M Sherman. 1987. A Prolog model of the income tax act of Canada. In Proceedings of the 1st international conference on Artificial intelligence and law. 127–136. find
  54. Edward H Shortliffe and Bruce G Buchanan. 1975. A model of inexact reasoning in medicine. Mathematical biosciences 23, 3-4 (1975), 351–379. ↗
  55. Jerrold Soh, How Khang Lim, and Ian Ernst Chai. 2019. Legal Area Classification: A Comparative Study of Text Classifiers on Singapore Supreme Court Judgments. Association for Computational Linguistics, Minneapolis, Minnesota. find
  56. Anne vdL Gardner. 1983. The design of a legal analysis program. In AAAI-83. 114–118. find
  57. Lai Dac Viet, Vu Trong Sinh, Nguyen Le Minh, and Ken Satoh. 2017. ConvAMR: Abstract meaning representation parsing for legal document. arXiv preprint arXiv:1711.06141 (2017). ↗
  58. Thiemo Wambsganß and Hansjörg Fromm. 2019. Mining User-Generated Repair Instructions from Automotive Web Communities. In Proceedings of the 52nd Hawaii International Conference on System Sciences. find
  59. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems. 3261–3275.
  60. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698 (2015). ↗
  61. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 2369–2380. https://doi.org/10.18653/v1/D18-1259 ↗
  62. Masaharu Yoshioka, Yoshinobu Kano, Naoki Kiyota, and Ken Satoh. 2018. Overview of japanese statute law retrieval and entailment task at coliee-2018. In Twelfth International Workshop on Juris-informatics (JURISIN 2018). find
  63. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326 (2018). ↗
  64. Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019. Broad-Coverage Semantic Parsing as Transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 3786–3798. https://doi.org/10.18653/v1/D19-1392 ↗
  65. Ziqi Zhang, Philip Webster, Victoria S Uren, Andrea Varga, and Fabio Ciravegna. 2012. Automatically Extracting Procedural Knowledge from Instructional Texts using Natural Language Processing.. In LREC, Vol. 2012. 520–527. find
  66. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2019. JEC-QA: A Legal-Domain Question Answering Dataset. arXiv preprint arXiv:1911.12011 (2019). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/holzenberger-2020-dataset-statutory-reasoning-tax-law/