Elamur Research
☰
home / papers / 2021 / CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review

CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2103.06268

Summary

Contract review is slow and costly legal work, and it has had almost no machine learning support. Part of the reason is that no large public dataset annotated by experts existed. The authors, working with the non-profit The Atticus Project, built CUAD (Contract Understanding Atticus Dataset). It contains more than 500 contracts labeled by trained law students and lawyers for 41 clause categories that lawyers care about, such as Governing Law, Anti-Assignment, or Perpetual License. Given a contract and a category, a model must locate the relevant passages, which are often a few sentences buried in very long documents. The authors benchmarked ten pretrained Transformer models. Results improve quickly with newer architectures but remain far from solved. The amount of labeled data turned out to matter much more than model size or domain-specific pretraining. The dataset gives legal NLP a demanding benchmark for specialized domains. It could also make contract review cheaper for small businesses and individuals.

Key points

  • Dataset scale: CUAD has over 13,000 expert annotations across 41 label categories, covering 9,283 pages.
    • Annotators completed 70–100 hours of training.
    • Three additional annotators checked every label.
    • The authors estimate the labeling effort is worth over $2 million.
  • Progress across model generations: Precision @ 80% Recall went from 8.2% for BERT (2018) to 44.0% for DeBERTa (2021). DeBERTa-xlarge was the strongest model, scoring 15.4 AUPR points above the weakest.
  • Model size has inconsistent effects:
    • BERT-base and BERT-large scored almost the same (32.4% vs. 32.3% AUPR).
    • The largest ALBERT model was over 20× bigger than the smallest but gained only about 3%.
    • RoBERTa-large did clearly better than RoBERTa-base (48.2% vs. 42.6%).
  • Data is the main bottleneck: For RoBERTa-base, ten times more training data raised AUPR from 27.6% to 42.6%. In contrast, extra masked-LM pretraining on about 8GB of EDGAR contracts added only about 3%.
  • Uneven performance by category: Per-category AUPR for DeBERTa-xlarge ranges from close to 100% down to roughly 20%. Data and code are at atticusprojectai.org/cuad and github.com/TheAtticusProject/cuad.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Law

Cited by

172 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 166 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2103.06268 · arXiv 2103.06268

@misc{hendrycks2021cuad,
  title = {{CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review}},
  author = {Hendrycks, Dan and Burns, Collin and Chen, Anya and Ball, Spencer},
  year = {2021},
  doi = {10.48550/arxiv.2103.06268},
  eprint = {2103.06268},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2103.06268}
}

References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ondrej Bojar, C. Buck, C. Federmann, B. Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and A. Tamchyna. Findings of the 2014 workshop on statistical machine translation. In WMT at ACL, 2014. find our card
  2. T. Brown, B. Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, G. Krüger, T. Henighan, R. Child, Aditya Ramesh, D. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, E. Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, J. Clark, Christopher Berner, Sam McCandlish, A. Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005.14165, 2020. ↗
  3. CEB. Advance your contract management process, 2017. URL https://web.archive.org/web/20170920135124/https://www.cebglobal.com/compliance-legal/smb-legal/contract-management-midsized.html. ↗
  4. Ilias Chalkidis, Ion Androutsopoulos, and A. Michos. Extracting contract elements. Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law, 2017. find
  5. Ilias Chalkidis, Ion Androutsopoulos, and A. Michos. Obligation and prohibition extraction using hierarchical rnns. ArXiv, abs/1805.03871, 2018. ↗
  6. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. Large-scale multi-label text classification on eu legislation. In ACL, 2019. find
  7. J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019. find
  8. X. Duan, Baoxin Wang, Ziyue Wang, Wentao Ma, Yiming Cui, D. Wu, S. Wang, T. Liu, Tianxiang Huo, Z. Hu, Heng Wang, and Z. Liu. Cjrc: A reliable human-annotated benchmark dataset for chinese judicial reading comprehension. ArXiv, abs/1912.09156, 2019. ↗
  9. Soham Gadgil, Mark Endo, Emily P. Wen, A. Ng, and P. Rajpurkar. Chexseg: Combining expert annotations with dnn-generated saliency maps for x-ray segmentation. ArXiv, 2021. find
  10. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. ArXiv, abs/2004.10964, 2020. ↗
  11. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv, abs/2006.03654, 2020. ↗ our card
  12. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021a. ↗ our card
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021b. find our card
  14. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021c. ↗ our card
  15. N. Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. A dataset for statutory reasoning in tax law entailment and question answering. In NLLP@KDD, 2020. find
  16. Yoshinobu Kano, Miyoung Kim, M. Yoshioka, Yao Lu, J. Rabelo, Naoki Kiyota, R. Goebel, and K. Satoh. Coliee-2018: Evaluation of the competition on legal information extraction and entailment. In JSAI-isAI Workshops, 2018. find
  17. Diederik P. Kingma and Jimmy Ba. abs/1412.6980, 2015. ↗
  18. Adam: A method for stochastic optimization. CoRR, Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942, 2020. ↗
  19. Spyretta Leivaditi, J. Rossi, and E. Kanoulas. A benchmark for lease contract review. ArXiv, abs/2010.10386, 2020. ↗
  20. Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019. ↗
  21. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. ArXiv, abs/1806.03822, 2018. ↗ our card
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017. ↗
  23. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019. find our card
  24. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, 2020. find our card
  25. Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. CVPR, 2015. find
  26. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. How does nlp benefit legal system: A summary of legal artificial intelligence. ArXiv, abs/2004.12158, 2020. ↗ our card