Elamur Research⌕
☰
home / papers / 2023 / SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval

SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval

Paper · 2023 · Haitao Li, Qingyao Ai, Jia Chen et al.

Summary

SAILER tackles legal case retrieval, where a query case must be matched to relevant precedents. General-purpose pre-trained retrievers struggle here for two reasons. Case documents are long and follow a fixed internal logic. Legal relevance also depends on small but decisive legal elements: buying and stealing the same goods can look nearly identical as text and still lead to opposite judgments. The authors split each case into five parts (Procedure, Fact, Reasoning, Decision, Tail). They pre-train an asymmetric encoder-decoder model on unlabeled case documents. A deep encoder turns the Fact section into a dense vector. Two lightweight decoders then use that vector to recover heavily masked text in the Reasoning and Decision sections. This pushes the Fact embedding to carry information that matters for the judgment. The model was tested on four public Chinese and English legal benchmarks, including LeCaRD, in zero-shot and fine-tuned settings. It beat prior state-of-the-art baselines in both. This shows that document structure can stand in for scarce expert relevance labels.

Key points

  • Cases are segmented into Procedure, Fact, Reasoning, Decision and Tail. The authors argue this layout holds for both Case Law (e.g., US) and Statute Law (e.g., China) documents.
  • Architecture: one deep encoder over the Fact part, plus two shallow decoders (a Reasoning Decoder and a Decision Decoder). Code is released at github.com/CSHaitao/SAILER.
  • On LeCaRD, the best setting used an encoder mask ratio of 0.15 and a decoder mask ratio of 0.45. Performance became unstable once decoder masking went above 0.45.
  • Ablations:
    • Dropping either decoder hurts.
    • Removing the Decision Decoder costs more than removing the Reasoning Decoder.
    • Removing all decoders causes a sharp drop.
    • Adding more decoder layers lowers performance, consistent with SEED.
  • A zero-shot t-SNE plot covered six easily confused charges, including Theft, Robbery, Affray and Provocation. SAILER separated them more cleanly than Chinese BERT, Chinese RoBERTa and SEED.

Interesting ideas

  • I1. Use one section to predict another. Instead of reconstructing the input itself, the model encodes one section (facts) and decodes a different section (reasoning or verdict). This forces the embedding to capture what drives the outcome rather than surface detail. The trick could carry over to any genre with a fixed layout, such as patents, medical reports or customs declarations.
  • I2. Asymmetric bottleneck. Pairing a strong encoder with weak, aggressively masked decoders means the decoders cannot reconstruct text on their own. They must rely on the dense vector, which makes the bottleneck do the real work. This is a cheap way to improve embeddings for dense retrieval.
  • I3. Several decoders, each with its own objective. Different pre-training signals are attached to one shared embedding through separate light decoders. Objectives can then be added or ablated independently.
  • I4. Label-free sanity checks. Plotting zero-shot embeddings of confusable classes with t-SNE, and comparing attention heatmaps against a baseline, shows whether a model has picked up domain-relevant distinctions before any fine-tuning. This is useful when annotated data is scarce.
  • I5. Masking position matters. The authors note that random masking may miss the key legal elements. This suggests that masking spans chosen by domain importance could strengthen similar pre-training setups.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🏢Huawei TechnologiesShenzhen · China96 works · 15,432 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksInformation Systems · 1,730 worksPolitical Science and International Relations · 206 works

🧭 Topics 2Research subjects and languages the work is about.Information retrieval & search · 552 worksLegal AI · 150 works

Cite

Cite as Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Yueyue Wu, Yiqun Liu, Chong Chen and Qi Tian. 2023. SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval. arXiv:2304.11370.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.11370 · arXiv 2304.11370

@misc{li2023sailer,
  title = {{SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval}},
  author = {Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Yueyue Wu, Yiqun Liu, Chong Chen and Qi Tian},
  year = {2023},
  doi = {10.48550/arxiv.2304.11370},
  eprint = {2304.11370},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.11370}
}

References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akiko Aizawa. 2003. An information-theoretic perspective of tf–idf measures. Information Processing & Management 39, 1 (2003), 45–65. ↗
  2. Sophia Althammer, Arian Askari, Suzan Verberne, and Allan Hanbury. 2021. DoSSIER@ COLIEE 2021: leveraging dense retrieval and summarization-based re-ranking for case law retrieval. arXiv preprint arXiv:2108.03937 (2021). ↗
  3. Trevor Bench-Capon, Michał Araszkiewicz, Kevin Ashley, Katie Atkinson, Floris Bex, Filipe Borges, Daniele Bourcier, Paul Bourgine, Jack G Conrad, Enrico Francesconi, et al. 2012. A history of AI and Law in 50 papers: 25 years of the international conference on AI and Law. Artificial Intelligence and Law 20, 3 (2012), 215–319. ↗
  4. Paheli Bhattacharya, Kripabandhu Ghosh, Arindam Pal, and Saptarshi Ghosh. 2022. Legal case document similarity: You need both network and text. Information Processing & Management 59, 6 (2022), 103069. ↗
  5. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. arXiv preprint arXiv:2010.02559 (2020). ↗
  6. Jia Chen, Haitao Li, Weihang Su, Qingyao Ai, and Yiqun Liu. [n. d.]. THUIR at WSDM Cup 2023 Task 1: Unbiased Learning to Rank. ([n. d.]). find
  7. Jia Chen, Yiqun Liu, Yan Fang, Jiaxin Mao, Hui Fang, Shenghao Yang, Xiaohui Xie, Min Zhang, and Shaoping Ma. 2022. Axiomatically Regularized Pre-training for Ad hoc Search. (2022). find
  8. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the TREC 2019 deep learning track. arXiv preprint arXiv:2003.07820 (2020). ↗
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). ↗
  10. Qian Dong, Yiding Liu, Suqi Cheng, Shuaiqiang Wang, Zhicong Cheng, Shuzi Niu, and Dawei Yin. 2022. Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking. arXiv preprint arXiv:2204.11673 (2022). ↗
  11. Qian Dong and Shuzi Niu. 2021. Legal judgment prediction via relational learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 983–992. find
  12. Yixing Fan, Xiaohui Xie, Yinqiong Cai, Jia Chen, Xinyu Ma, Xiangsheng Li, Ruqing Zhang, Jiafeng Guo, et al. 2022. Pre-training methods in information retrieval. Foundations and Trends® in Information Retrieval 16, 3 (2022), 178–317. ↗
  13. Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines 30, 4 (2020), 681–694. ↗
  14. Luyu Gao and Jamie Callan. 2021. Condenser: a pre-training architecture for dense retrieval. arXiv preprint arXiv:2104.08253 (2021). ↗
  15. Luyu Gao and Jamie Callan. 2021. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108.05540 (2021). ↗
  16. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 (2020). ↗
  17. Haitao Li, Jia Chen, Weihang Su, Qingyao Ai, and Yiqun Liu. 2023. Towards Better Web Search Performance: Pre-training, Fine-tuning and Learning to Rank. arXiv preprint arXiv:2303.04710 (2023). ↗
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). ↗
  19. Zheng Liu and Yingxia Shao. 2022. RetroMAE: Pre-training Retrieval-oriented Transformers via Masked Auto-Encoder. arXiv preprint arXiv:2205.12035 (2022). ↗
  20. Ilya Loshchilov and Frank Hutter. 2018. Fixing weight decay regularization in adam. (2018). find
  21. Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tieyan Liu, and Arnold Overwijk. 2021. Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder. arXiv preprint arXiv:2102.09206 (2021). ↗
  22. Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, and Xueqi Cheng. 2022. Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction. arXiv preprint arXiv:2204.10641 (2022). ↗
  23. Yixiao Ma, Yunqiu Shao, Bulou Liu, Yiqun Liu, Min Zhang, and Shaoping Ma. 2021. Retrieving legal cases from a large-scale candidate corpus. Proceedings of the Eighth International Competition on Legal Information Extraction/Entailment, COLIEE2021 (2021). find
  24. Yixiao Ma, Yunqiu Shao, Yueyue Wu, Yiqun Liu, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2021. LeCaRD: a legal case retrieval dataset for Chinese law system. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2342–2348. find
  25. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPs.
  26. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 5835–5847. ↗
  27. Juliano Rabelo, Randy Goebel, Mi-Young Kim, Yoshinobu Kano, Masaharu Yoshioka, and Ken Satoh. 2022. Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021. The Review of Socionetwork Strategies 16, 1 (2022), 111–133. ↗
  28. Juliano Rabelo, Mi-Young Kim, Randy Goebel, Masaharu Yoshioka, Yoshinobu Kano, and Ken Satoh. 2020. COLIEE 2020: methods for legal document retrieval and entailment. In JSAI International Symposium on Artificial Intelligence. Springer, 196–210. find
  29. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389. ↗
  30. Manavalan Saravanan, Balaraman Ravindran, and Shivani Raman. 2009. Improving legal information retrieval using an ontological framework. Artificial Intelligence and Law 17, 2 (2009), 101–124. ↗
  31. Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Ken Satoh, Min Zhang, and Shaoping Ma. 2020. BERT-PLI: Modeling Paragraph-Level Interactions for Legal Case Retrieval.. In IJCAI. 3501–3507. find
  32. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Simlm: Pre-training with representation bottleneck for dense passage retrieval. arXiv preprint arXiv:2207.02578 (2022). ↗
  33. Xing Wu, Guangyuan Ma, Meng Lin, Zijia Lin, Zhongyuan Wang, and Songlin Hu. 2022. Contextual mask auto-encoder for dense passage retrieval. arXiv preprint arXiv:2208.07670 (2022). ↗
  34. Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun. 2021. Lawformer: A pre-trained language model for chinese legal long documents. AI Open 2 (2021), 79–84. ↗
  35. Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yiqun Liu, et al. 2023. T2Ranking: A large-scale Chinese Benchmark for Passage Ranking. arXiv preprint arXiv:2304.03679 (2023). ↗
  36. Shenghao Yang, Haitao Li, Zhumin Chu, Jingtao Zhan, Yiqun Liu, Min Zhang, and Shaoping Ma. 2022. THUIR at the NTCIR-16 WWW-4 Task. Proceedings of NTCIR-16. to appear (2022). find
  37. Shenghao Yang, Yiqun Liu, Xiaohui Xie, Min Zhang, and Shaoping Ma. 2023. Enhance Performance of Ad-hoc Search via Prompt Learning. In Information Retrieval: 28th China Conference, CCIR 2022, Chongqing, China, September 16–18, 2022, Revised Selected Papers. Springer, 28–39. find
  38. Weijie Yu, Zhongxiang Sun, Jun Xu, Zhenhua Dong, Xu Chen, Hongteng Xu, and Ji-Rong Wen. 2022. Explainable legal case matching via inverse optimal transport-based rationale extraction. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 657–668. find
  39. Yiming Zeng, Ruili Wang, John Zeleznikow, and Elizabeth Kemp. 2005. Knowledge representation for the intelligent legal case retrieval. In International Conference on Knowledge-Based and Intelligent Information and Engineering Systems. Springer, 339–345. find
  40. ChengXiang Zhai. 2008. Statistical language models for information retrieval. Synthesis lectures on human language technologies 1, 1 (2008), 1–141. ↗
  41. Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing dense retrieval model training with hard negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1503–1512. find
  42. Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2020. Rep-BERT: Contextualized text embeddings for first-stage retrieval. arXiv preprint arXiv:2006.15498 (2020). ↗
  43. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129 (2019). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2023-sailer-structure-aware-pre-trained/