Elamur Research⌕
☰
home / papers / 2026 / Fine-Grained Semantic Comparison of Legal Documents using LLMs

Fine-Grained Semantic Comparison of Legal Documents using LLMs

Paper · 2026 · Elisei Rykov, Nikolay Ivanov, Maria Bandulevich et al.

Summary

Legal and regulatory documents in large organizations are revised often, and these revisions can introduce contradictions that reviewers miss. Existing comparison tools work at the character level, so they fail on paraphrases and meaning shifts. The authors address paragraph-level semantic comparison of legal texts. Instead of whole sentences, they compare smaller spans and label each one as Contradiction, Addition or Equivalent.

They introduce LegDiff, a benchmark made up of pairs of Russian paragraphs that were labeled by hand. The pairs come from real temporal revisions of Russian Codes, sourced from Consultant+. To discourage reliance on surface overlap, the Equivalent spans were paraphrased by an LLM and then checked by humans. The authors also provide:

  • LegDiff-Synth, a synthetic data pipeline that applies span-level edit operations,
  • a small English counterpart, LegDiff-en,
  • a tool for visualizing detected differences.

Experiments compare a difflib baseline with open and proprietary LLMs. Character-level diffing is adequate only for near-identical texts. Once paraphrasing is involved, proprietary LLMs perform best. Fine-tuning on the synthetic data substantially improves results and transfers across domains. The work matters because it offers a reproducible resource for interpretable semantic diffing in legal review.

Key points

  • LegDiff annotation was done by two Russian-proficient annotators, with raw agreement of 71.2% before adjudication. A moderator then resolved the disagreements. Equivalent spans were aligned automatically and then paraphrased with Claude Opus-4.6.
  • The LegDiff-Synth pipeline segments paragraphs into spans and samples edits for each span: paraphrases for Equivalent, edits for Contradiction, and inserted Addition placeholders. A human audit of a 10% sample rated about 90% of the samples as good.
  • LegDiff-en contains 44 English paragraph pairs with 78 Equivalent, 40 Addition and 31 Contradiction spans.
    • On its non-paraphrased version, difflib reaches an average of 66.92, ahead of GPT-5.2 (64.88) and Claude Opus 4.6 (63.26).
    • Small open models such as Llama3.1-8B and Qwen2.5-7B/32B score far lower.
    • In the paraphrased setting, GPT-5.2 is best overall, while Claude Opus 4.6 is slightly better on Equivalent spans.
  • In the threshold analysis for GPT-5.2 on paraphrased LegDiff, the average falls from 71.76 at threshold 25 to 33.01 at threshold 99. At the chosen τ = 0.8, the average is 51.68. Equivalent is the class most sensitive to the threshold.
  • Data, code and prompts are released at github.com/s-nlp/SLeDoC. All LLMs were evaluated with greedy decoding at temperature 0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Research University Higher School of EconomicsMoscow · Russia24 works · 39,585 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPolitical Science and International Relations · 206 works

🧭 Topics 1Research subjects and languages the work is about.Legal AI · 150 works

Cite

Cite as Elisei Rykov, Nikolay Ivanov, Maria Bandulevich, Kseniia Petrushina, Valentin Malykh, Vasily Konovalov, Alexander Panchenko and Ilseyar Alimova. 2026. Fine-Grained Semantic Comparison of Legal Documents using LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.86

@inproceedings{rykov2026fine,
  title = {{Fine-Grained Semantic Comparison of Legal Documents using LLMs}},
  author = {Elisei Rykov, Nikolay Ivanov, Maria Bandulevich, Kseniia Petrushina, Valentin Malykh, Vasily Konovalov, Alexander Panchenko and Ilseyar Alimova},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.86},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.86}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dor Bernsohn, Gil Semo, Yaron Vazana, Gila Hayat, Ben Hagag, Joel Niklaus, Rohit Saha, and Kyryl Truskovskyi. 2024. Legallens: Leveraging llms for legal violation identification in unstructured text. arXiv preprint arXiv:2402.04335. ↗
  2. Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Dan Roth, and Tal Schuster. 2023. Propsegment: A large-scale corpus for proposition-level segmentation and entailment recognition. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 8874–8893. Association for Computational Linguistics.
  3. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  4. Jyun-Yu Jiang, Mingyang Zhang, Cheng Li, Michael Bendersky, Nadav Golbandi, and Marc Najork. 2019. Semantic text matching for long-form documents. In The World Wide Web Conference, WWW ’19, page 795–806, New York, NY, USA. Association for Computing Machinery. find
  5. A Khasianov, I Alimova, A Marchenko, G Nurham-betova, E Tutubalina, and D Zuev. 2018. Lawyer’s intellectual tool for analysis of legal documents in russian. In 2018 International conference on artificial intelligence applications and innovations (IC-AIAI), pages 42–46. IEEE. find
  6. Yuta Koreeda and Christopher Manning. 2021. ContractNLI: A dataset for document-level natural language inference for contracts. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1907–1919, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  7. Barrett Lattimer, Patrick H. Chen, Xinyuan Zhang, and Yi Yang. 2023. Fast and accurate factual inconsistency detection over long documents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1691–1703, Singapore. Association for Computational Linguistics. text corpora. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM ’17, pages 325–334, New York, NY, USA. Association for Computing Machinery. find
  8. Lucas E. Resck, Jean R. Ponciano, Luis Gustavo Nonato, and Jorge Poco. 2023. Legalvis: Exploring and inferring precedent citations in legal documents. IEEE Trans. Vis. Comput. Graph., 29(6):3105–3120. ↗
  9. Elisei Rykov, Nikolay Ivanov, Kseniia Petrushina, Maria Bandulevich, Valentin Malykh, Vasily Konovalov, Alexander Panchenko, and Ilseyar Alimova. 2026. Sledoc: System for legal document comparison. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), pages 1–6, Melbourne, VIC, Australia. ACM. find
  10. Gerrit Schumann and Jorge Marx Gómex. 2024. Detection of conflicts, contradictions and inconsistencies in regulatory documents: a literature review. In 2024 Fifth International Conference on Intelligent Data Science Technologies and Applications (IDSTA), pages 81–88. IEEE. find
  11. Gerrit Schumann, Jorge Marx Gómez, and Hergen Pargmann. 2026. Detection of contradictions and inconsistencies in regulatory documents using prompt-engineering. find
  12. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  13. Mengying Yuan, Wangzi Xuan, and Fei Li. 2025. Cross-document cross-lingual natural language inference via rst-enhanced graph fusion and interpretability prediction. CoRR, abs/2504.12324. A The segmentation results are shown in Table 8. In the fine-tuned setting, Qwen2.5-Instruct achieves the best performance on PropSegmEnt, outperforming all few-shot baselines on both Jaccard and Exact Match. ↗
  14. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. arXiv preprint arXiv:2509.06888. ↗
  15. Sagnik Ray Choudhury, Pepa Atanasova, and Isabelle Augenstein. 2023. Explaining interactions between text spans. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12709–12730, Singapore. Association for Computational Linguistics. PropSegmEnt Segmentation Task Results Method (params) Mode Jaccard Exact T5-Large (0.7B) Qwen2.5-Instruct (32B) Qwen2.5-Instruct (32B) GPT-4o (-) FT FS FT FS 55.50 37.91 61.69 40.21 32.28 27.59 37.96 27.05 *Table 8: Segmentation performance on PropSegmEnt. We report Jaccard-based span match and Exact Match. FS denotes few-shot prompting and FT denotes fine-tuning. T5-Large scores are taken from the original PropSegmEnt paper.* Xiang Ren, Yuanhua Lv, Kuansan Wang, and Jiawei Han. 2017. Comparative document analysis for large find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rykov-2026-fine-grained-semantic-comparison-legal/