Elamur Research
☰
home / papers / 2020 / LEGAL-BERT: The Muppets straight out of Law School

LEGAL-BERT: The Muppets straight out of Law School

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.findings-emnlp.261.pdf

Summary

General-purpose BERT models are trained on text such as Wikipedia and books. They tend to lose accuracy on specialised language, and legal text is a clear example, with its distinctive vocabulary, formal syntax and reliance on domain knowledge. The authors compare three ways of bringing BERT into the legal field:

  • using the original model unchanged;
  • continuing its pre-training on legal material (FP);
  • training a new model from scratch with its own sub-word vocabulary (SC).

For the domain-specific variants, they assembled about 12 GB of English legal documents, including legislation, court decisions and contracts. They also tested a compact model and widened the hyper-parameter grid used for fine-tuning beyond the usual recommendations of Devlin et al. On three legal benchmarks, the domain-adapted models nearly always beat a carefully tuned BERT-BASE. The largest gains came on the hardest tasks, where in-domain knowledge matters most. The compact variant performed close to the full-size models at a fraction of the cost. The resulting LEGAL-BERT family is publicly released, giving legal NLP and legal-tech work ready-made domain models and practical advice for adapting BERT to other specialised fields.

Key points

  • Pre-training corpus: six sources totalling about 12 GB.
    • EU and UK legislation
    • ECJ and ECHR cases
    • US court cases (Case Law Access Project)
    • US contracts (SEC-EDGAR)
  • Models released: LEGAL-BERT-FP, LEGAL-BERT-SC and LEGAL-BERT-SMALL, available at huggingface.co/nlpaueb.
    • LEGAL-BERT-FP continues training BERT-BASE for up to 500k steps.
    • LEGAL-BERT-SC has 110M parameters and a new sentencepiece vocabulary.
    • LEGAL-BERT-SMALL has 6 layers, 512 hidden units, 8 heads and 35M parameters, and runs roughly 4× faster.
  • Training setup: the from-scratch models were trained for 1M steps with batch size 256 and sequences of up to 512 tokens, on Cloud TPU v3.
  • Evaluation datasets: EURLEX57K (multi-label classification), ECHR-CASES (binary and multi-label classification) and CONTRACTS-NER (three subsets).
    • Gains were about +2.5% on ECHR multi-label, +1.8% on contract header and +1.6% on dispute resolution.
    • The gain on EURLEX57K was only about +0.2%.
  • Fine-tuning: an enlarged search space clearly improved results on most tasks. It added batch sizes 4–32, a learning rate of 1e-5, dropout of 0.2, and early stopping in place of a fixed epoch limit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Law

Cited by

496 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 482 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.261 · arXiv 2010.02559

@inproceedings{chalkidis2020legal,
  title = {{LEGAL-BERT: The Muppets straight out of Law School}},
  author = {Chalkidis, Ilias and Fergadiotis, Manos and Malakasiotis, Prodromos and Aletras, Nikolaos and Androutsopoulos, Ion},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.261},
  eprint = {2010.02559},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.261}
}

References

8 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. I. Chalkidis, I. Androutsopoulos, and A. Michos. 2017. Extracting Contract Elements. In Proceedings of the International Conference of AI and Law, London, UK. find
  2. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. find our card
  3. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  4. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. CoRR, abs/1909.11942. ↗
  5. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. In CoRR. find our card
  6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.11692. ↗
  7. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. ↗ our card
  8. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480–1489. Association for Computational Linguistics. find