Elamur Research⌕
☰
home / papers / 2020 / LEDGAR: A Large-Scale Multi-label Corpus for Text Classification of Legal Provisions in Contracts

LEDGAR: A Large-Scale Multi-label Corpus for Text Classification of Legal Provisions in Contracts

Summary

Legal NLP has few openly available labeled resources, and before this work no public corpus of contract provisions annotated by topic existed. The authors built LEDGAR from material contracts (Exhibit-10 filings) that companies submit to the U.S. SEC through EDGAR. They exploited a formatting habit: many provisions begin with a bold or underlined heading that names their type. Treating that heading as a label and the text after it as the provision, they harvested labeled examples semi-automatically. They then cleaned the data with heuristics: de-duplication, splitting compound labels, merging singular and plural forms, and removing rare or narrowly distributed labels. The final corpus has 846,274 provisions from 60,540 contracts, more than 100m tokens, and 12,608 labels, with about 16% of provisions carrying several labels. The paper also discusses ways to sample subcorpora, proposes deriving a label hierarchy from label names, compares several classifiers, and tests how well they transfer to contracts from outside the corpus. LEDGAR is useful for legal studies, domain-specific NLP, and large-scale or extreme multilabel classification.

Key points

  • Source and scale: all Exhibit-10 contracts filed from 2016 to 2019 were crawled, giving 117,578 contracts. The first scrape produced 1,850,284 labeled provisions in 72,605 contracts, with 183,622 distinct labels.
  • Cleanup sequence: de-duplication cut provisions to 1,081,177. Label splitting, singular/plural merging, dropping labels found in fewer than five contracts, and outlier removal then produced the final 12,608 labels and 846,274 provisions. The abstract gives smaller rounded figures that do not match these table counts.
  • Minimum-frequency thresholds shrink the labelset sharply. Keeping labels with at least 100 occurrences leaves 1,442 labels, at least 1,000 leaves 130, and at least 10,000 leaves only 5. The released corpus applies no threshold, so users choose their own.
  • The share of multilabel provisions rose from about 6% to about 16–17% once ambiguous compound labels were split.
  • The paper defines an NDA-oriented subcorpus with a curated set of provision types, such as Governing Law, Severability, Non-Assignment, and Confidential Information.

Interesting ideas

  • I1. Labels from document layout. Text that is visually marked as a heading (bold or underlined, followed by a period or colon) can act as free supervision for the paragraph that follows. This could produce labeled data from any collection of structured documents, such as regulations, standards, or official templates.
  • I2. Cautious splitting of compound labels. Delimiters like "and", commas, or "&" can either join two labels or belong to a single label. The authors split only when each resulting part already exists as a standalone label. This cheap check is a reusable way to normalize noisy, user-generated tag sets.
  • I3. Regression-based outlier filtering. The authors fit a linear relation between how often a label occurs and how many documents it appears in. They then remove labels that fall well below that line, meaning labels concentrated in only a few documents. The distance is normalized by the expected value, so frequent labels are allowed more variance. The same approach could flag boilerplate or source-specific categories in other crawled datasets.
  • I4. Release raw, filter downstream. The corpus is published without a frequency cutoff, and tables show how different thresholds change its size. Users can then build anything from extreme-classification benchmarks to small, practical labelsets from the same resource.
  • I5. Testing outside the source corpus. The classifiers are also evaluated on contracts that did not come from the corpus. This shows whether models trained on cheaply scraped data hold up on real documents, which is worth checking for any automatically built training set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Law

Cited by

68 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{tuggener2020ledgar,
  title = {{LEDGAR: A Large-Scale Multi-label Corpus for Text Classification of Legal Provisions in Contracts}},
  author = {Tuggener, Don and Däniken, Pius von and Peetz, Thomas and Cieliebak, Mark},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  doi = {10.21256/zhaw-20087},
  url = {https://doi.org/10.21256/zhaw-20087}
}

References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikolaos Aletras, et al., editors. (2019). Proceedings of the Natural Legal Language Processing Workshop 2019, Minneapolis, Minnesota. Association for Computational Linguistics. find
  2. Bengio, S., Dembczynski, K., Joachims, T., Kloft, M., and Varma, M. (2019). Extreme Classification (Dagstuhl Seminar 18291). Dagstuhl Reports, 8(7):62–80. ↗
  3. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  4. Bommarito, M. J., Katz, D. M., and Detterman, E. M. (2018a). LexNLP: Natural language processing and information extraction for legal and regulatory texts. arXiv preprint, abs/1806.03688. ↗
  5. Bommarito, M. J., Katz, D. M., and Detterman, E. M. (2018b). OpenEDGAR: Open source software for SEC EDGAR analysis. arXiv preprint, abs/1806.04973. ↗
  6. Chalkidis, I., Fergadiotis, E., Malakasiotis, P., and Androutsopoulos, I. (2019). Large-scale multi-label text classification on EU legislation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6314–6322, Florence, Italy, July. Association for Computational Linguistics. find
  7. Choromanska, A. and Kumar Jain, I. (2019). Extreme multiclass classification criteria. Computation, 7(1). ↗
  8. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  9. Fisch, J. E. (2018). Governance by contract: The implications for corporate bylaws. Calif. L. Rev., 106:373. ↗
  10. Glaser, I., Scepankova, E., and Matthes, F. (2018). Classifying semantic types of legal sentences: Portability of machine learning models. In JURIX, pages 61–70. ↗
  11. Hershkoff, H. and Kahan, M. (2018). Selection provisions in corporate contracts. Wash. L. Rev., 93:265. find
  12. Jain, H., Balasubramanian, V., Chunduri, B., and Varma, M. (2019). Slice: Scalable linear extreme classifiers trained on 100 million labels for related searches. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 528–536. ACM. find
  13. Joulin, A., Bojanowski, P., Mikolov, T., Jégou, H., and Grave, E. (2018). Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find our card
  14. Jeroen Keppens et al., editors. (2017). Proceedings of the 16th edition of the International Conference on Artificial Intelligence and Law, ICAIL 2017. ACM. find
  15. Knapp, C. L., Crystal, N. M., and Prince, H. G. (2019). Problems in Contract Law: cases and materials. Aspen Publishers. find
  16. Mills, K. R. (2019). Talking about regulation in 10-k annual reports; uniformity in a naive sample. Finance Graduate Theses & Dissertations. find
  17. Monica Palmirani, editor. (2018). Legal Knowledge and Information Systems - JURIX 2018: The Thirty-first Annual Conference, Groningen, The Netherlands, 12-14 December 2018, volume 313 of Frontiers in Artificial Intelligence and Applications. IOS Press. find
  18. Georg Rehm, et al., editors. (2018). Proceedings of the 1st Workshop on Language Resources and Technologies for the Legal Knowledge Graph, Paris, France. European Language Resources Association (ELRA). find
  19. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. find
  20. Soni, A., Pappu, A., Busa-Fekete, R., and Dembczynski, K. (2018). Extreme multilabel classification for social media chairs’ welcome and organization. In Companion Proceedings of the The Web Conference 2018, pages 1893–1894. International World Wide Web Conferences Steering Committee. find
  21. Trosborg, A. (1991). An analysis of legal speech acts in english contract law. ’It is hereby performed’. HERMES-Journal of Language and Communication in Business, 4(6):65–90. find
  22. Trosborg, A. (1995). Statutes and contracts: An analysis of legal speech acts in the english language of the law. Journal of Pragmatics, 23(1):31–53. ↗
  23. Waltl, B., Muhr, J., Glaser, I., Bonczek, G., Scepankova, E., and Matthes, F. (2017). Classifying legal norms with active machine learning. Legal Knowledge and Information Systems, page 11. ↗
  24. Waltl, B., Bonczek, G., Scepankova, E., and Matthes, F. (2019). Semantic types of legal norms in german laws: classification and analysis using local linear explanations. Artificial Intelligence and Law, 27(1):43–71. ↗
  25. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019). Transformers: State-of-the-art natural language processing. find our card
  26. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., and Hovy, E. (2016). Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480–1489. find our card
  27. Yovel, J. (2000). What is contract law about–speech act theory and a critique of skeletal promises. Northwestern University Law Review, 94:937. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tuggener-2020-ledgar-large-scale-multi-label/