Elamur Research⌕
☰
home / papers / 2026 / RegTrack: A Fine-Grained Benchmark for Multi-Class Legal Change Detection

RegTrack: A Fine-Grained Benchmark for Multi-Class Legal Change Detection

Paper · 2026 · Joe Yu, Kevin Chenhao Li and Julian Ostarek

Summary

Regulations change over time, and compliance teams need to know whether an amendment alters legal obligations or only fixes wording. Existing diff tools compare surface text and cannot make that distinction. The authors (TU Munich) built RegTrack, an English benchmark from EUR-Lex. It covers consecutive consolidated versions of EU regulations in the environmental, transport and emissions domains. The texts are split into Atomic Legal Units (ALUs), meaning articles, paragraphs or points. Units are aligned automatically, pre-labelled with rule-based heuristics, and then checked by human annotators with a mediator resolving disagreements. Each pair receives one of six change labels.

The benchmark defines three tasks: alignment, classification of already aligned pairs, and an end-to-end combined task. Lexical methods, dense encoders and zero-shot LLMs serve as baselines. LLMs are strongest when classifying pre-aligned pairs. Their weaker alignment, however, pulls down their end-to-end scores, where general-purpose sentence encoders perform best. The resource supports regulatory monitoring and version-aware legal retrieval.

Key points

  • Dataset size: 4,772 annotated ALU pairs drawn from 6 EU regulations and 18 consolidated versions.
  • Label taxonomy and distribution: no_change (74.29%), addition (14.19%), semantic_change (5.57%), deletion (4.25%), syntactic_change (1.49%) and reference_update (0.21%).
  • Annotation reliability: 0.9707 simple agreement on alignment and Cohen's κ of 0.9332 on change labels.
  • Main results:
    • DeepSeek-V3.2 reaches 0.916 Macro-F1 on classification but only 0.568 on the combined task.
    • paraphrase-multilingual-mpnet-base-v2 has the best combined Macro-F1 (0.600).
    • all-MiniLM-L6-v2 has the best combined accuracy (0.919).
    • DocuToads remains strong on alignment (0.781).
  • Limitations:
    • Annexes are excluded.
    • Classes are heavily imbalanced.
    • LLMs may have seen EUR-Lex texts during pre-training (possible contamination).
    • A German label transfer was tested only as an auxiliary analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technical University of MunichMunich · Germany55 works · 48,474 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksPolitical Science and International Relations · 182 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksLegal AI · 134 works

Cite

Cite as Joe Yu, Kevin Chenhao Li and Julian Ostarek. 2026. RegTrack: A Fine-Grained Benchmark for Multi-Class Legal Change Detection. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.68

@inproceedings{yu2026regtrack,
  title = {{RegTrack: A Fine-Grained Benchmark for Multi-Class Legal Change Detection}},
  author = {Joe Yu, Kevin Chenhao Li and Julian Ostarek},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.68},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.68}
}

References

33 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sallam Abualhaija, Marcello Ceci, Nicolas Sannier, Domenico Bianculli, Lionel C. Briand, Dirk Zetzsche, and Marco Bodellini. 2024. Ai-enabled regulatory change analysis of legal requirements. In 2024 IEEE 32nd International Requirements Engineering Conference (RE), pages 5–17. find
  2. Wolfgang Alschner. 2020. Sense and similarity: automating legal text comparison. In Computational Legal Studies, pages 9–28. Edward Elgar Publishing. ↗
  3. Farid Ariai, Joel Mackenzie, and Gianluca Demartini. 2025. Natural language processing for the legal domain: A survey of tasks, datasets, models, and challenges. ACM Computing Surveys, 58(6):1–37. ↗
  4. Max Bachmann. 2024. rapidfuzz/rapidfuzz: Release 3.6.2. find
  5. Julien Breton, Mokhtar Mokhtar Billami, Max Chevalier, Ha Thanh Nguyen, Ken Satoh, Cassia Trojahn, and May Myo Zin. 2025. Leveraging llms for legal terms extraction with limited annotated data. Artificial Intelligence and Law, pages 1–27. ↗
  6. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  7. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904, Online. Association for Computational Linguistics. ↗
  8. Pierre-André Côté. 1991. The interpretation of legislation in Canada. Cownsville, Qu�ebec: Editions Y. Blais. find
  9. Hudson de Martim. 2025a. Modeling the diachronic evolution of legal norms: An lrmoo-based, component-level, event-centric approach to legal knowledge graphs. arXiv preprint arXiv:2506.07853. ↗
  10. Henrik Hermansson and James P Cross. 2016. Tracking amendments to legislation and other political texts with a novel minimum-edit-distance algorithm: Docutoads. arXiv preprint arXiv:1608.06459. ↗
  11. Hudson de Martim. 2025b. An ontology-driven graph rag for legal norms: A structural, temporal, and deterministic approach. arXiv preprint arXiv:2505.00039. ↗
  12. Daniel Huwiler, Kurt Stockinger, and Jonathan Fürst. 2025. Versionrag: Version-aware retrieval-augmented generation for evolving documents. arXiv preprint arXiv:2510.08109. ↗
  13. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. Deepseek-v3.2: Pushing the frontier of open large language models. Preprint, arXiv:2512.02556. ↗
  14. Daniel Martin Katz, Corinna Coupette, Janis Beckedorf, and Dirk Hartung. 2020. Complex societies and the growth of the law. Scientific reports, 10(1):18737. ↗
  15. VI Lcvenshtcin. 1966. Binary coors capable or ‘correcting deletions, insertions, and reversals. In Soviet physics-doklady, volume 10. find
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  17. Janet Erasmus. 1998. Plain language drafting meets interpretive principles and rules–a drafter’s perspective. Canadian Institute for the Administration of Justice. find
  18. European Parliament and Council of the European Union and European Commission. 2015. Joint practical guide for persons involved in the drafting of European Union legislation. Publications Office of the European Union. find
  19. European Union. 2026. EUR-Lex: Access to European Union law. Accessed: 2026-02-23. find
  20. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 2288–2292, New York, NY, USA. Association for Computing Machinery. ↗
  21. Daniel Fürst, Mennatallah El-Assady, Daniel A Keim, and Maximilian T Fischer. 2025. Challenges and opportunities for visual analytics in jurisprudence. Artificial Intelligence and Law, pages 1–32. ↗
  22. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  23. Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. In Advances in Neural Information Processing Systems, volume 35, pages 29217–29234. Curran Associates, Inc. find
  24. Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119–9130, Online. Association for Computational Linguistics.
  25. Xiang Li, Jiaxun Gao, Diana Inkpen, and Wolfgang Alschner. 2022. Detecting relevant differences between similar legal texts. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 256–264, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  26. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925. ↗
  27. Publications Office of the European Union. 2026a. Cellar: The EU common content and metadata repository and SPARQL endpoint. https://publications.europa.eu/ en/web/about-us/legal-notices/cellar. Data retrieved via SPARQL endpoint: https://publications.europa.eu/webapi/ rdf/sparql. ↗
  28. Publications Office of the European Union. 2026b. Interinstitutional style guide. Accessed: February 23, 2026. find
  29. Python Software Foundation. 2026. difflib — helpers for computing deltas. https://docs.python.org/3/library/difflib. html#difflib.SequenceMatcher. Accessed: 2026-02-24. ↗
  30. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  31. Marco Siino, Mariana Falco, Daniele Croce, and Paolo Rosso. 2025. Exploring llms applications in law: A literature review on current legal nlp approaches. IEEE Access, 13:18253–18276. Deletion An existing ALU removed from the new version. A.3 ↗
  32. Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316. The Compliance Impact Rule If a manufacturer must alter their behavior or reporting due to this change, it is a Modification. If they can continue existing operations exactly as before, it is a Rewording. ↗
  33. Xinzhe Xu, Liang Zhao, Hongshen Xu, and Chen Chen. 2025. Claw: Benchmarking chinese legal knowledge in large language models-a fine-grained corpus and reasoning analysis. arXiv preprint arXiv:2509.21208. B Representative Error Cases for Existing Systems Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, and 1 others. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176. This appendix provides two representative ALU pairs where existing semantic change detection systems are likely to fail: one false positive (semantic over-prediction) and one false negative (semantic under-prediction). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2026-regtrack-fine-grained-benchmark-multi/