Elamur Research⌕
☰
home / papers / 2021 / When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset

When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset

Paper · 2021 · Lucia Zheng, Neel Guha, Brandon R. Anderson et al.

Summary

Earlier attempts to pretrain Transformers on legal text gave little or no improvement over general-purpose BERT. This is puzzling, since lawyers widely regard legal language as distinctive. The authors argue that the benchmarks used so far were too easy or poorly matched to the pretraining domain. To test this, they built CaseHOLD, a benchmark of 53,000+ multiple-choice items drawn from U.S. case law from 1965 onward. Each item asks the model to pick the correct holding of a cited case. The items are generated automatically from citation conventions: the parenthetical holding statements that accompany citations serve as gold answers, and similar but wrong holdings serve as distractors. They then pretrained Legal-BERT on roughly 3.5M U.S. court decisions with a custom legal vocabulary and compared it with BiLSTM and standard BERT on CaseHOLD and two existing legal tasks. Legal-BERT gave its largest gain on CaseHOLD and smaller gains elsewhere. The size of the gain followed how domain-specific each task was. The work offers practical guidance on when costly domain pretraining is worth doing.

Key points

  • CaseHOLD format: each item has a citing-text prompt and five candidate holding statements, one correct and four plausible but wrong. The prompt uses a fixed context window that may begin mid-sentence.
  • Baseline difficulty: BiLSTM reaches F1 ≈ 0.4 and general BERT about 0.6. Law students and lawyers reportedly solve the task with high accuracy.
  • Legal-BERT gain: Legal-BERT was pretrained on ≈3.5M decisions from all U.S. courts (a corpus larger than BERT's own) with a custom vocabulary. It improves CaseHOLD F1 by 7.2 points, about 12% relative to BERT. This is the largest pretraining gain reported for a legal task and is comparable to those of SciBERT and BioBERT.
  • Existing tasks are easier: on Overruling, a BiLSTM already reaches F1 0.91. Gains on Overruling and Terms of Service are smaller but consistent, and the authors report the best result so far on the general Terms of Service setting.
  • When gains are largest: besides domain specificity, the size of the fine-tuning set matters most. Gains are biggest when labeled data is scarce. Code, datasets and models are released at github.com/reglab/casehold.

Interesting ideas

  • I1. Use domain conventions as free labels: the legal citation system provides supervision without attorney annotation. Other fields with structured referencing, such as patents, standards or legislation, could yield benchmarks in the same way.
  • I2. Choose hard distractors: wrong answers are picked to be semantically close to the correct holding, so the multiple-choice task cannot be solved through surface cues. This is a general recipe for building discriminative benchmarks from paired texts.
  • I3. Estimate domain-specificity before pretraining: the authors use a domain-specificity (DS) score to judge whether a task is in-domain enough to benefit. This matters because full pretraining can cost over $1M, and the same check could be applied with SciBERT, BioBERT or a Ukrainian-domain model.
  • I4. Vary training-set size and prompt difficulty: running these controlled variants shows where domain pretraining helps, including zero-shot gains. It is a cheap way to map the regimes in which a specialized model is worth its cost.
  • I5. Check for memorization: the authors note that gains may partly come from recalling case names seen during pretraining. They suggest paraphrased-holding variants to separate memorization from real reasoning, a useful check for any benchmark built from corpus-adjacent text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPolitical Science and International Relations · 206 worksLaw · 84 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksLegal AI · 150 works

Cite

Cite as Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson and Daniel E. Ho. 2021. When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset. arXiv:2104.08671.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2104.08671 · arXiv 2104.08671

@misc{zheng2021when,
  title = {{When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset}},
  author = {Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson and Daniel E. Ho},
  year = {2021},
  doi = {10.48550/arxiv.2104.08671},
  eprint = {2104.08671},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2104.08671}
}

References

50 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preoţiuc-Pietro, and Vasileios Lampos. 2016. Predicting judicial decisions of the European Court of Human Rights: A natural language processing perspective. PeerJ Computer Science 2 (2016), e93.
  2. Pablo D Arredondo. 2017. Harvesting and Utilizing Explanatory Parentheticals. SCL Rev. 69 (2017), 659. find
  3. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A Pretrained Language Model for Scientific Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 3615–3620.
  4. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, New York, NY, USA, 610–623. find
  5. Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural Legal Judgment Prediction in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 4317–4323. https://www.aclweb.org/anthology/P19-1424 ↗
  6. Ilias Chalkidis, Ion Androutsopoulos, and Achilleas Michos. 2017. Extracting Contract Elements. In Proceedings of the 16th Edition of the International Conference on Articial Intelligence and Law (London, United Kingdom) (ICAIL ’17). Association for Computing Machinery, New York, NY, USA, 19–28. find
  7. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The Muppets straight out of Law School. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 2898–2904. https://www.aclweb.org/anthology/2020.findings-emnlp.261 ↗
  8. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2019. Neural Contract Element Extraction Revisited. Workshop on Document Intelligence at NeurIPS 2019. https://openreview.net/forum?id= B1x6fa95UH ↗
  9. Columbia Law Review Ass’n, Harvard Law Review Ass’n, and Yale Law Journal. 2015. The Bluebook: A Uniform System of Citation (21st ed.). The Columbia Law Review, The Harvard Law Review, The University of Pennsylvania Law Review, and The Yale Law Journal. find
  10. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-Training with Whole Word Masking for Chinese BERT. arXiv:1906.08101 [cs.CL] ↗
  11. Laura C. Dabney. 2008. Citators: Past, Present, and Future. Legal Reference Services Quarterly 27, 2-3 (2008), 165–190. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://www.aclweb.org/anthology/N19-1423 ↗
  13. Pintip Hompluem Dunn. 2003. How judges overrule: Speech act theory and the doctrine of stare decisis. Yale LJ 113 (2003), 493. find
  14. Emad Elwany, Dave Moore, and Gaurav Oberoi. 2019. BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding. arXiv:1911.00473 http://arxiv.org/abs/1911.00473 ↗
  15. David Freeman Engstrom and Daniel E Ho. 2020. Algorithmic accountability in the administrative state. Yale J. on Reg. 37 (2020), 800. find
  16. David Freeman Engstrom, Daniel E. Ho, Catherine Sharkey, and Mariano-Florentino Cuéllar. 2020. Government by Algorithm: Artificial Intelligence in Federal Administrative Agencies. Administrative Conference of the United States, Washington DC, United States. find
  17. European Union 1993. Council Directive 93/13/EEC of 5 April 1993 on unfair terms in consumer contracts. European Union. find
  18. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning AI With Shared Human Values. arXiv:2008.02275 [cs.CY] ↗
  19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. arXiv:2009.03300 [cs.CY] ↗
  20. Michael J. Bommarito II, Daniel Martin Katz, and Eric M. Detterman. 2018. LexNLP: Natural language processing and information extraction for legal and regulatory texts. arXiv:1806.03688 http://arxiv.org/abs/1806.03688 ↗
  21. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics 8 (2020), 64–77. https://www.aclweb.org/anthology/2020.tacl-1.5 ↗
  22. David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018. Measuring the Evolution of a Scientific Field through Citation Frames. Transactions of the Association for Computational Linguistics 6 (2018), 391–406. https://www.aclweb.org/anthology/Q18-1028 ↗
  23. Minki Kang, Moonsu Han, and Sung Ju Hwang. 2020. Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 6102– 6120. https://www.aclweb.org/anthology/2020.emnlp-main.493 ↗
  24. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. arXiv:1808.06226 [cs.CL] ↗
  25. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 4 (2019), 1234–1240.
  26. Marco Lippi, Przemysław Pałka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni. 2019. CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law 27, 2 (2019), 117–139.
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs.CL] ↗
  28. David Mellinkoff. 2004. The language of the law. Wipf and Stock Publishers, Eugene, Oregon. find
  29. Elizabeth Mertz. 2007. The Language of Law School: Learning to “Think Like a Lawyer”. Oxford University Press, USA. find
  30. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. http://arxiv.org/abs/1301. 3781 ↗
  31. Octavia-Maria, Marcos Zampieri, Shervin Malmasi, Mihaela Vela, Liviu P. Dinu, and Josef van Genabith. 2017. Exploring the Use of Text Classification in the Legal Domain. Proceedings of 2nd Workshop on Automated Semantic Analysis of Information in Legal Texts (ASAIL). find
  32. Adam R. Pah, David L. Schwartz, Sarath Sanga, Zachary D. Clopton, Peter DiCola, Rachel Davis Mersey, Charlotte S. Alexander, Kristian J. Hammond, and Luís A. Nunes Amaral. 2020. How to build a more open justice system. Science 369, 6500 (2020), 134–136. find
  33. Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrQA: A Large Corpus for Question Answering on Electronic Medical Records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 2357– 2368. https://www.aclweb.org/anthology/D18-1258 ↗
  34. Marc Queudot, Éric Charton, and Marie-Jean Meurs. 2020. Improving Access to Justice with Legal Chatbots. Stats 3, 3 (2020), 356–375. find
  35. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  36. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, 2383–2392. https://www.aclweb.org/anthology/D16-1264 ↗
  37. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics 7 (2019), 249–266.
  38. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2021. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics 8 (2021), 842–866.
  39. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2020. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. arXiv:2012.15613 [cs.CL] ↗
  40. Jaromir Savelka, Vern R Walker, Matthias Grabmair, and Kevin D Ashley. 2017. Sentence boundary detection in adjudicatory decisions in the United States. Traitement automatique des langues 58 (2017), 21. find
  41. Or Sharir, Barak Peleg, and Yoav Shoham. 2020. The Cost of Training NLP Models: A Concise Overview. arXiv:2004.08900 [cs.CL] ↗
  42. Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2020. Unnatural Language Inference. arXiv:2101.00010 [cs.CL] ↗
  43. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: Enhanced Representation through Knowledge Integration. arXiv:1904.09223 [cs.CL] ↗
  44. P.M. Tiersma. 1999. Legal Language. University of Chicago Press, Chicago, Illinois. https://books.google.com/books?id=Sq8XXTo3A48C ↗
  45. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artiéres, Axel-Cyrille Ngonga Ngomo, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. 2015. An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics 16, 1 (April 2015), 138. find
  46. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, Brussels, Belgium, 353–355. https://www.aclweb.org/anthology/W18-5446 ↗
  47. Jonah Wu. 2019. AI Goes to Court: The Growing Landscape of AI for Access to Justice. https://medium.com/legal-design-and-innovation/ai-goes-to-court-the-growing-landscape-of-ai-for-access-to-justice-3f58aca4306f ↗
  48. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv:1609.08144 [cs.CL] ↗
  49. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5218–5230. https://www.aclweb.org/anthology/2020.acl-main.466 ↗
  50. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. JEC-QA: A Legal-Domain Question Answering Dataset. , 9701-9708 pages. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2021-pretraining-help-assessing-self-supervised/