Elamur Research⌕
☰
home / papers / 2022 / AraLegal-BERT: A pretrained language model for Arabic Legal text

AraLegal-BERT: A pretrained language model for Arabic Legal text

Paper · 2022 · Muhammad Al-qurishi, Sarah Alqaseemi and Riad Souissi

Summary

General-purpose Arabic BERT models are trained mostly on everyday Modern Standard Arabic, and they handle the specialised wording of legal texts poorly. The authors address this by training a BERT model from scratch only on Arabic legal material. They collected about 4.5 GB of public documents, mostly Saudi and Gulf court cases, plus theses, Fiqh books, laws and contracts. After deduplication this gave roughly 13.7 million sentences. Preprocessing followed the AraBERT recipe, including segmentation of words into prefix, stem and suffix, and pretraining used standard masked language modeling.

The fine-tuned base model, AraLegal-BERT, was compared with Arabertv2-Large, ARBERT and mBERT on three tasks: legal text classification on two datasets (BoG and SJP), named entity recognition and keyword extraction. The 12-layer model beat all baselines, including a larger general model. Its advantage was modest on classification and very large on the two token-level tasks.

The work matters because it shows that a compact in-domain model can outperform heavier generic models in a terminology-heavy niche. It also costs less memory and runs faster.

Key points

  • Pretraining corpus: 4.5 GB of public Arabic legal text, about 13.7M sentences after deduplication. Table 1 lists the source types as Books (6K), Cases (336K), Terms and laws (3K) and others (5K).

  • Vocabulary and objective: a vocabulary of about 64,000 entries built on prefix/stem/suffix segmentation (Abdelali et al., 2016). Pretraining used MLM with the usual 15% selection split into 80% masked, 10% random and 10% unchanged.

  • Training setup: about 50 epochs and around 500K steps on an NVIDIA DGX1 with 8 GPUs. Effective batch size was 512, maximum sequence length 512, and learning rate between 1e-5 and 5e-5. Both a 12-layer and a 24-layer model were pretrained, but only the base model was fine-tuned and evaluated.

  • Classification results (macro F1):

    DatasetAraLegal-BERT (base)Arabertv2-LargemBERT
    BoG0.8910.8270.598
    SJP0.9220.8850.783

    The prose describes smaller margins than these tables show.

  • Token-level results: macro F1 was about 21% higher than Arabertv2-Large on legal NER (BIO tags) and about 26% higher than ARBERT on keyword extraction. The general models performed especially poorly on recall.

Interesting ideas

  • I1. Splitting words into prefix, stem and suffix before building the subword vocabulary makes the units match the language's morphology, so the model learns from meaningful pieces rather than arbitrary fragments.
  • I2. When an input exceeded the model window, retaining its opening 128 tokens together with its closing 382 tokens outperformed other truncation schemes, because the beginning and end of a document carry the most document-level signal.
  • I3. Only a plain linear layer was put on top of the encoder for tagging, so performance differences can be credited to pretraining rather than to a clever task head.
  • I4. The generic models failed mainly on recall in domain tagging, not on precision, which points to vocabulary coverage rather than wrong judgments as the cause.
  • I5. A smaller model trained purely in-domain beat a larger general one, suggesting that matching the domain can matter more than raw capacity in narrow fields.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Explicitly separating English affixes (un-, -ness, -ed, -ly) before lemmatisation would make each morphological decision traceable to a named affix.
  • I2 (answer): When a reader must classify or summarise a long text under a size budget, it could prioritise the opening and closing sentences before the middle.
  • I3 (tree): Probing gparse encoder variants with a fixed linear head would show cleanly which pretraining data actually improves dependency labels.
  • I4 (learn): Evaluation could report per-domain recall separately from precision, flagging domains where silver-data witnesses agree only because they all miss rare domain terms.
  • I5 (world): Compiling small domain-specific lexicons (legal, school-science) on top of the general WordNet/Wikidata ontology may pay off more than enlarging the general knowledge base.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksLegal AI · 134 works

Cite

Cite as Muhammad Al-qurishi, Sarah Alqaseemi and Riad Souissi. 2022. AraLegal-BERT: A pretrained language model for Arabic Legal text. Proceedings of the Natural Legal Language Processing Workshop 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.nllp-1.31

@inproceedings{alqurishi2022aralegal,
  title = {{AraLegal-BERT: A pretrained language model for Arabic Legal text}},
  author = {Muhammad Al-qurishi, Sarah Alqaseemi and Riad Souissi},
  booktitle = {Proceedings of the Natural Legal Language Processing Workshop 2022},
  year = {2022},
  doi = {10.18653/v1/2022.nllp-1.31},
  url = {https://doi.org/10.18653/v1/2022.nllp-1.31}
}

References

21 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16.
  2. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. arXiv preprint arXiv:2102.10684. ↗
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  4. Muhammad Saleh Al-Qurishi and Riad Souissi. 2021. Arabic named entity recognition using transformer-based-crf model. In Proceedings of The Fourth International Conference on Natural Language and Speech Processing (ICNLSP 2021), pages 262–271.
  5. Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323. ↗
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  7. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676. ↗
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  9. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559. ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. *Table 4: Overall results of all fine-tuned models in the Keywords Extraction Task* Model / Macro-Average Precision Recall F1-score Arabertv2-Large (Antoun et al., 2020) ARBERT (Abdul-Mageed et al., 2020) mBERT (Devlin et al., 2018) AraLegal-BERT (base) 0.765979 0.789460 0.721356 0.93481 0.470098 0.524715 0.375075 0.84449 0.582625 0.630420 0.493533 0.88736 *Table 5: Overall results of all fine-tuned models in the Named Entity Recognition Task* Model / Macro-Average Precision Recall F1-score Arabertv2-Large (Antoun et al., 2020) ARBERT (Abdul-Mageed et al., 2020) mBERT (Devlin et al., 2018) AraLegal-BERT (base) 0.891934 0.889916 0.886825 0.89848 0.450073 0.413266 0.326475 0.73644 0.598261 0.564423 0.477254 0.80943 ↗
  11. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. ↗
  12. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842–866. ↗
  13. Raquel Silveira, CG Fernandes, João A Monteiro Neto, Vasco Furtado, and José Ernesto Pimentel Filho. 2021. Topic modelling of legal documents via legal-bert. Proceedings http://ceur-ws org ISSN, 1613:0073. language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45. ↗
  14. Chin Man Yeung. 2019. Effects of inserting domain vocabulary and fine-tuning bert for german legal language. Master’s thesis, University of Twente. find
  15. Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018. Recent trends in deep learning based natural language processing. ieee Computational intelligenCe magazine, 13(3):55–75. find
  16. Gechuan Zhang, Paul Nulty, and David Lillis. 2022. Enhancing legal argument mining with domain pre-training and neural networks. arXiv preprint arXiv:2202.13457. ↗
  17. Xing Su, Shan Xue, Fanzhen Liu, Jia Wu, Jian Yang, Chuan Zhou, Wenbin Hu, Cecile Paris, Surya Nepal, Di Jin, et al. 2022. A comprehensive survey on community detection with deep learning. IEEE Transactions on Neural Networks and Learning Systems. ↗
  18. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In China national conference on Chinese computational linguistics, pages 194–206. Springer. ↗
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  20. Sabine Wehnert, Shipra Dureja, Libin Kutty, Viju Sudhi, and Ernesto William De Luca. 2022. Applying bert embeddings to predict legal textual entailment. The Review of Socionetwork Strategies, 16(1):197–219. ↗
  21. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qurishi-2022-aralegal-bert-pretrained-language-model/