Elamur Research⌕
☰
home / papers / 2026 / ViLegalLM: Language Models for Vietnamese Legal Text

ViLegalLM: Language Models for Vietnamese Legal Text

Paper · 2026 · Truong-Phuc Nguyen, Quy-Nhan Nguyen and Minh-Tien Nguyen

Summary

Vietnamese legal NLP has had few dedicated pretrained models. Earlier options were trained on small corpora (0.32–2.52GB), and some were adapted from instruction-tuned checkpoints, which risks losing capabilities. The authors present ViLegalLM, a set of three models continually pretrained on a new 16GB Vietnamese legal corpus collected from four public legal repositories. The suite includes ViLegalBERT, an encoder trained with MLM, and two decoders trained with CLM: ViLegalQwen2.5-1.5B-Base and ViLegalQwen3-1.7B-Base. Labeled legal data is scarce, so they also built three synthetic datasets for True/False QA, multiple-choice QA and NLI. These were created with LLM generation and hard negative mining, then checked by human annotators. Evaluated on ten benchmarks covering retrieval, question answering, NLI and syllogism reasoning, the models reach the best results among open-source systems. The study also finds that adapting base checkpoints works better than adapting instruction-tuned ones. All resources are released publicly, giving the community a strong baseline for this under-resourced domain.

Key points

  • Corpus construction: 1.39M crawled pages from TVPL, VBPL, LuatVietnam and LawNet yielded 1.14M parsed documents (19GB). FastText language filtering (confidence ≥ 0.95) cut this to 18GB, and MinHashLSH near-duplicate removal (threshold 0.8) brought it to 16GB.
  • Models:
    • ViLegalBERT: 135M parameters, initialized from PhoBERT-base-v2, 256-token context, input word-segmented with PyVi.
    • ViLegalQwen2.5-1.5B-Base: 1.54B parameters, 2,048-token context.
    • ViLegalQwen3-1.7B-Base: 1.72B parameters, 4,096-token context.
  • Synthetic datasets: ViLegalTF (13,032 training items), ViLegalMCQ (14,920 training items) and ViLegalNLI. All are grounded in the ALQAC legal corpus and verified by five annotators.
  • Benchmarks: ALQAC-IR, ZALO-IR, ALQAC-TF/MCQ/EQA, ViBidLQA-EQA/AQA, VLSP-MCQ-LK, VLSP-NLI and VLSP-Syllogism. Error analysis shows that inaccurate statute citations are the main source of syllogism reasoning failures.
  • Compute and limitations: training took 144, 168 and 24 GPU hours (A100/H100) for the three models. Limitations include a maximum size of 1.7B parameters, the 256-token encoder context, and no significance testing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hung Yen University of Technology and EducationHưng Yên · Vietnam2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 1Research subjects and languages the work is about.Legal AI · 134 works

Cite

Cite as Truong-Phuc Nguyen, Quy-Nhan Nguyen and Minh-Tien Nguyen. 2026. ViLegalLM: Language Models for Vietnamese Legal Text. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1801

@inproceedings{nguyen2026vilegallm,
  title = {{ViLegalLM: Language Models for Vietnamese Legal Text}},
  author = {Truong-Phuc Nguyen, Quy-Nhan Nguyen and Minh-Tien Nguyen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1801},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1801}
}

References

48 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. We address several ethical considerations relevant to developing and deploying Vietnamese legal pre-trained language models. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21–29. IEEE. Intended Use and Limitations ViLegalLM is designed to support legal NLP research in Vietnamese, including document retrieval, question answering, and text analysis. These models are not intended to replace professional legal counsel; users should not rely solely on model outputs for consequential legal decisions without qualified professional consultation. Data and Privacy Our 16GB pretraining corpus was collected exclusively from four publicly available Vietnamese legal repositories (TVPL, VBPL, LuatVietnam, LawNet), containing only official legal texts and publicly accessible materials. find
  2. No Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898– 2904, Online. Association for Computational Linguistics.
  3. Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Katz, and Anders Søgaard. 2023. LeXFiles and LegalLAMA: Facilitating English multinational legal language model development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15513–15535, Toronto, Canada. Association for Computational Linguistics.
  4. Chieu-Nguyen Chau, Truong-Son Nguyen, and Le-Minh Nguyen. 2020. Vnlawbert: A vietnamese legal answer selection approach using bert language model. In 2020 7th NAFOSTED Conference on Information and Computer Science (NICS), pages 298– 301. IEEE. find
  5. Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre FT Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, and 1 others. 2024. Saullm-7b: A pioneering large language model for law. arXiv preprint arXiv:2403.03883. ↗
  6. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. ↗
  8. Dinh-Truong Do, Son T Luu, Trang Pham, Trung Vo, Nguyen-Hoang Chu, Quang-Huy Chu, Cuong Nguyen, Minh Nguyen, An Trieu, Dat Nguyen, and 1 others. 2024. A summary of the alqac 2024 competition. In 2024 16th International Conference on Knowledge and System Engineering (KSE), pages 422–427. IEEE. find
  9. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. 2024. LawBench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7933–7962, Miami, Florida, USA. Association for Computational Linguistics. ↗
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  11. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in neural information processing systems, 36:44123–44279.
  12. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. ↗
  13. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nlp dataset for legal contract review. Preprint, arXiv:2103.06268. ↗
  14. Wanhong Huang, Yi Feng, Chuanyi Li, Honghan Wu, Jidong Ge, and Vincent Ng. 2024. CMDL: A large-scale Chinese multi-defendant legal judgment prediction dataset. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5895–5906, Bangkok, Thailand. Association for Computational Linguistics. find
  15. Piotr Indyk and Rajeev Motwani. 1998. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 604–613. find
  16. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446. ↗
  17. Ishan Jindal, Chandana Badrinath, Pranjal Bharti, Lakkidi Vinay, and Sachin Dev Sharma. 2024. Balancing continuous pre-training and instruction fine-tuning: Optimizing instruction-following in llms. Preprint, arXiv:2410.10739. ↗
  18. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Association for Computational Linguistics. find
  19. Kamal raj Kanakarajan, Bhuvana Kundumani, and Malaikannan Sankarasubbu. 2021. BioELEC-TRA:pretrained biomedical text encoder using discriminators. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 143–154, Online. Association for Computational Linguistics. find
  20. Anh-Cuong Le, Trong-Chi Duong, Viet-Ha Nguyen, and Thang VQ Le. 2025. Overview of the Legal-SLM shared task: Evaluating legal reasoning of Vietnamese small language models. In Proceedings of the 11th International Workshop on Vietnamese Language and Speech Processing, pages 147–152, Hanoi, Vietnam. Association for Computational Linguistics. find
  21. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. ↗
  22. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  23. Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Dhagash Mehta, Stefano Pasquali, Wei Cheng, Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, and 5 others. 2025. Domain specialization as the key to make large language models disruptive: A comprehensive survey. ACM Comput. Surv., 58(3). find
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  25. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. ↗
  26. Nguyen Minh, Vu Hoang Tran, Vu Hoang, Huy Duc Ta, Trung Huu Bui, and Steven Quoc Hung Truong. 2022. ViHealthBERT: Pre-trained language models for Vietnamese in health text mining. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 328–337, Marseille, France. European Language Resources Association. find
  27. Dat Quoc Nguyen, Linh The Nguyen, Chi Tran, Dung Ngoc Nguyen, Dinh Phung, and Hung Bui. 2023a. PhoGPT: Generative Pre-training for Vietnamese. arXiv preprint, arXiv:2311.02945. ↗
  28. Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1037–1042, Online. Association for Computational Linguistics.
  29. Nam Nguyen, Thang Phan, Duc-Vu Nguyen, and Kiet Nguyen. 2023b. ViSoBERT: A pre-trained language model for Vietnamese social media text processing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5191–5207, Singapore. Association for Computational Linguistics. ↗
  30. Thu-Ha Nguyen, Truong-Phuc Nguyen, and Khang Tran.T. 2025a. Vibidlqa: Vietnamese bidding legal question answering dataset. find
  31. Truong-Phuc Nguyen, Quy-Nhan Nguyen, Manh-Cuong Phan, Chi-Hai Cao, Trinh-Hoai-An Duong, and Minh-Tien Nguyen. 2025b. Utehy-nlu@alqac 2025: Dynamic weighted ensemble and adaptive reasoning for vietnamese legal text processing. In 2025 17th International Conference on Knowledge and System Engineering (KSE), pages 1–5. find
  32. Nguyen Van Huy Nguyen Nho Trung, Nguyen Nhat Quang. 2025. Vietnamese_embedding: Embedding model in vietnamese language. find
  33. Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel Ho. 2024. MultiLegalPile: A 689GB multilingual legal corpus. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15077–15094, Bangkok, Thailand. Association for Computational Linguistics. ↗
  34. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  35. Nhu Hai Phung, Chi Thanh Nguyen, Minh-Tien Nguyen, Thu Ha Nguyen, Huu Loi Le, and Truong-Phuc Nguyen. 2025. A fine-tuning framework based on question, context, and answer relationships for enhancing legal information retrieval. Engineering Applications of Artificial Intelligence, 159:111570. ↗
  36. Tran Minh Quang, Nguyen Xuan Phi, Nguyen Van Tai, Phan Minh Toan, and Dang Van Thin. 2025. Bosch@AI_Team at LegalSML 2025: Vietnamese legal small language with domain adaptation and aspect-based data synthesis. In Proceedings of the 11th International Workshop on Vietnamese Language and Speech Processing, pages 153–164, Hanoi, Vietnam. Association for Computational Linguistics. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. Improving language understanding by generative pre-training. find
  37. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. ↗
  38. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  39. Cong Dao Tran, Nhut Huy Pham, Anh Tuan Nguyen, Truong Son Hy, and Tu Vu. 2023. ViDeBERTa: A powerful pre-trained language model for Vietnamese. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1071–1078, Dubrovnik, Croatia. Association for Computational Linguistics. find
  40. Manh Tran-Tien, Huu-Loi Le, Dang Nhat Minh, T. Tran Khang, Huy-The Vu, and Nguyen Minh-Tien. 2023. ViPubmedDeBERTa: A pre-trained model for Vietnamese biomedical text. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, pages 831–840, Hong Kong, China. Association for Computational Linguistics. find
  41. Huy Trieu, Dang-Phuong-Nam Doan, Anh-Kiet Nguyen, Thanh-Thai Nguyen, Thanh-Nghia Vo, Tung Le, and Huy Tien Nguyen. 2025. Flame reavers@alqac 2025: Integrating learned rankers and llm reasoning in a dynamic hybrid architecture for legal retrieval. In 2025 17th International Conference on Knowledge and System Engineering (KSE), pages 1–6. find
  42. Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun. 2021. Lawformer: A pre-trained language model for chinese legal long documents. AI Open, 2:79–84. ↗
  43. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. ↗
  44. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 40 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. ↗
  45. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. ↗
  46. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623. Curran Associates, Inc.
  47. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5218–5230, Online. Association for Computational Linguistics.
  48. Zhi Zhou, Jiang-Xin Shi, Peng-Xiao Song, Xiao-Wen Yang, Yi-Xuan Jin, Lan-Zhe Guo, and Yu-Feng Li. 2024. Lawgpt: A chinese legal knowledge-enhanced large language model. Preprint, arXiv:2406.04614. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2026-vilegallm-language-models-vietnamese-legal/