Elamur Research⌕
☰
home / papers / 2026 / Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classification

Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classification

Paper · 2026 · Amrita Singh, H. Suhan Karaca, Aditya Joshi et al.

Summary

Legal NLP studies on contract classification usually benchmark general-purpose Transformers and seldom test models pretrained on legal text. This paper fills that gap with a systematic comparison. The authors fine-tuned 13 open-source legal-specific models and 9 generalist models on three English contract datasets:

  • UNFAIR-ToS: multi-label detection of unfair Terms of Service clauses.
  • LEDGAR: 100-way topic labelling of SEC provisions.
  • LEXDEMOD: multi-label deontic modality tied to contract parties.

The legal-specific set covers mostly base-size encoders plus PoL-BERT, two legal decoders and one encoder-decoder. Results are reported as micro-F1 and macro-F1. Domain-pretrained base models matched or beat the generalists and set new state-of-the-art results on UNFAIR-ToS and LEXDEMOD, although they are roughly a third the size of RoBERTa-large. Rare clause types benefited most. Encoders also handled long-tail labels better than billion-parameter decoders. Older models pretrained on focused contract and legal corpora beat newer ones trained on broad legal mixtures. The authors conclude that in-domain pretraining data matters more than scale. They argue that practitioners should consider compact legal encoders before turning to large generalist LLMs.

Key points

  • On UNFAIR-ToS, Contracts-BERT (110M) is the paper's best legal-specific model, with 96.2 µ-F1 and 83.4 m-F1. RoBERTa-large (355M) scores 95.8 and 81.6. On LEXDEMOD, Legal-BERT reaches 81.23 µ-F1 and 78.01 m-F1, against 77.88 µ-F1 for RoBERTa-large.
  • LEDGAR (60k/10k/10k split) is the exception. RoBERTa-large stays best there with 88.6 µ-F1 and 83.6 m-F1, while Legal-BERT is on par with the base-size generalists.
  • Averaged over all three tasks, Legal-BERT has the top mean µ-F1 (88.48) and Contracts-BERT the top mean m-F1 (81.10). Together with CaseLaw-BERT and LexLM, they are recommended as strong baselines.
  • Error analysis shows that domain pretraining fixes misclassifications of rare classes. On UNFAIR-ToS, Contracts-BERT correctly labels Limitation of Liability and Unilateral Termination clauses that RoBERTa-large misses. On LEDGAR, SaulLM improves over Mistral, and AdaptLLM over Llama-3.2.
  • Legal-BERT and Contracts-BERT were pretrained on only about 354k and 76k documents, yet they outperform newer genre-diverse legal models. PoL-BERT, for example, drops to 41.35 µ-F1 on LEXDEMOD.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓UNSW SydneySydney · Australia34 works · 1,509 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksLegal AI · 134 works

Cite

Cite as Amrita Singh, H. Suhan Karaca, Aditya Joshi, Hye-young Paik and Jiaojiao Jiang. 2026. Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classification. Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.customnlp4u-1.5

@inproceedings{singh2026evaluating,
  title = {{Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classification}},
  author = {Amrita Singh, H. Suhan Karaca, Aditya Joshi, Hye-young Paik and Jiaojiao Jiang},
  booktitle = {Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)},
  year = {2026},
  doi = {10.18653/v1/2026.customnlp4u-1.5},
  url = {https://doi.org/10.18653/v1/2026.customnlp4u-1.5}
}

References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. EU consumer protection law. Directive 93/13/eec on unfair terms in consumer contracts, article 3. http://data.europa.eu/eli/dir/1993/13/oj. Accessed on 24 July 2025. ↗
  2. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. ↗
  3. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. ↗
  4. Rohan Bhambhoria, Samuel Dahan, Jonathan Li, and Xiaodan Zhu. 2024. Evaluating ai for law: Bridging the gap with open-source solutions. arXiv preprint arXiv:2404.12349. ↗
  5. Momojit Biswas, Anmol Singhal, and Preethu Rose Anish. 2025. Optimized domain-specific text processing with keyword knowledge distillation (kkd). In 2025 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. find
  6. Saibo Geng, Rémi Lebret, and Karl Aberer. 2021. Legal transformer models may not always help. arXiv preprint arXiv:2109.06862. ↗
  7. V Sanh. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of Thirty-third Conference on Neural Information Processing Systems (NIPS2019). find
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2025. The llama 3 herd of models.
  9. Andrew Shin and Kunitake Kaneko. 2025. Improved understanding of legal text with graph attention networks. In Proceedings of the Twentieth International Conference on Artificial Intelligence and Law, pages 364–368. find
  10. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in neural information processing systems, 36:44123–44279.
  11. Amrita Singh, Aditya Joshi, Jiaojiao Jiang, and Hye-young Paik. 2025. A survey of classification tasks and approaches for legal contracts. Springer AI Review. ↗
  12. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations. ↗
  13. Amrita Singh, Preethu Rose Anish, Aparna Verma, Sivanthy Venkatesan, Logamurugan V, and Smita Ghaisas. 2024. A data decomposition-based hierarchical classification method for multi-label classification of contractual obligations for the purpose of their governance. Scientific Reports, 14(1):12755. ↗
  14. Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. Advances in Neural Information Processing Systems, 35:29217–29234. find
  15. Don Tuggener, Pius Von Däniken, Thomas Peetz, and Mark Cieliebak. 2020. Ledgar: a large-scale multi-label corpus for text classification of legal provisions in contracts. In Proceedings of the twelfth language resources and evaluation conference, pages 1235– 1241.
  16. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. ↗
  17. Santosh T.y.s.s, Cornelius Weiss, and Matthias Grabmair. 2024. LexSumm and LexT5: Benchmarking and modeling legal summarization tasks in English. In Proceedings of the Natural Legal Language Processing Workshop 2024, pages 381–403, Miami, FL, USA. Association for Computational Linguistics. ↗
  18. Marco Lippi, Przemysław Pałka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni. 2019. Claudette: an automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law, 27(2):117–139. ↗
  19. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. CoRR. ↗
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  21. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and 1 others. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297.
  22. Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel Ho. 2024. Multilegalpile: A 689gb multilingual legal corpus. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15077–15094. ↗
  23. Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021. When does pre-training help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the eighteenth international conference on artificial intelligence and law, pages 159–168. find
  24. Shounak Paul, Arpan Mandal, Pawan Goyal, and Saptarshi Ghosh. 2023. Pre-trained language models for the legal domain: A case study on indian law. In Proceedings of 19th International Conference on Artificial Intelligence and Law - ICAIL 2023. ↗
  25. A Abhilasha Sancheti, Aparna Garimella, Balaji Vasan Srinivasan, and Rachel Rudinger. 2022. Agent-specific deontic modality detection in legal language. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11563–11579. Dataset Statistics and Illustrative Examples This appendix provides an overview of the datasets used for benchmarking legal-specific models in Table 3 and presents labeled examples from all three datasets to aid understanding. The Dataset Contract Type Terms of Ser-UNFAIR-vice (Consumer ToS Contract) (Chalkidis et al., 2022) LEDGAR Exhibit-10 Ma-(Chalkidis terial Contract et al., 2022) LEXDEMOD Lease Contract (Sancheti et al., 2022) Task Task Type Train /Dev/ Test Class Unfair Contractual Terms Identification Multi-label Classification 5,532/2,275/ 1,607 Contract Provision Topic Classification Multi-class Classification 60,000/10,000/ 10,000 Agent-Specific Deontic Modality Detection Multi-label Classification 4,282/330/ 1,777 *Table 3: Overview of Datasets used for Benchmarking Legal-specific Models.* Legal-RoBERTa Legal-RoBERTa (Geng et al., 2021) builds on the RoBERTa-base model (125M parameters) and continues pre-training on 4.9 GB of legal text, including patent litigation documents, US court cases, and publicly available Google Patents data. find
  26. CaseLaw-BERT CaseLaw-BERT (Zheng et al., 2021) is a BERT-base-uncased model (110M parameters) pre-trained on 3.4M US federal and state court decisions from the Harvard Case Law corpus. Although originally referred to as Custom Legal-BERT by (Zheng et al., 2021), it is later termed CaseLaw-BERT by (Chalkidis et al., 2022) to distinguish it from the earlier Legal-BERT of (Chalkidis et al., 2020), highlighting its exclusive training on harvard case law. This naming convention is now widely adopted, and we follow the same in this work. find
  27. PoL-BERT PoL-BERT (Henderson et al., 2022) is a RoBERTa-large model (340M parameters) pre-trained on the Pile-of-Law, a 256GB corpus comprising 10M legal and administrative documents. The dataset spans a wide range of legal domains, including US federal and state court opinions (e.g., CourtListener, SCOTUS filings), regulatory documents (e.g., Federal Register, Code of Federal Regulations, SEC and IRS guidance), legislative texts (e.g., US Bills, US Code, State Codes), and other legal document sources (e.g., ECHR, Eur-Lex, ICJ/PCIJ rulings). It also includes administrative decisions from US agencies (e.g., DOJ, OLC, BVA, NLRB, EOIR, DOL), legal contracts (e.g., EDGAR filings, Atticus contracts, CFPB agreements), educational materials (e.g., open-access casebooks, exam outlines), and publicly available community-driven legal discussions. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2026-evaluating-customized-vs-generalist-transformer/