Elamur Research
☰
home / papers / 2021 / Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases

Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.naacl-main.22.pdf

Summary

Systems that predict court outcomes rarely explain their decisions, which undermines trust in a high-stakes field like law. This paper proposes a new task, alleged violation prediction, using European Court of Human Rights (ECtHR) judgments. Given a case's facts, a model must predict which ECHR articles the applicant claims were breached, which is a skewed multi-label problem. It must also select the fact paragraphs that justify its prediction.

The authors release a dataset of about 11k English ECtHR cases. It includes silver paragraph rationales, derived from the facts the court cites in its assessment, and gold rationales from legal experts on a test subset. They fine-tune a hierarchical BERT model (HIERBERT-HA) end-to-end and test existing rationale constraints as regularizers: sparsity, continuity and comprehensiveness. They also adapt comprehensiveness to the multi-label setting and add a new constraint, singularity. Continuity turns out to be unhelpful for paragraph-level selection. Singularity yields the most faithful and highest-quality rationales, beating even training with silver rationale supervision. Still, all models remain far from human-chosen rationales, so the benchmark leaves substantial room for future work.

Key points

  • Dataset. The released dataset (ecthr_cases on Hugging Face) splits into 9K train, 1K development and 1K test cases. Silver rationales cover 24–31% of paragraphs on average, and each case has about 1.7–1.8 allegations.
  • Singularity constraint. Singularity is the authors' new regularizer. It pushes the model toward a rationale mask that outperforms randomly sampled alternative masks.
  • Rationale quality results. The 50 test cases with both silver and gold rationales were evaluated with HIERBERT-HA and the singularity loss:
    • Against silver rationales, it scored mRP 45.1 and F1 40.9, versus 30.2 and 27.8 for a random baseline.
    • Against gold rationales, it scored mRP 53.6 and F1 48.3, versus 35.1 and 30.2 for random.
  • Silver versus gold rationales. Silver rationales are incomplete and partly noisy. However, they rank the methods in the same order as gold rationales do, so they could serve to evaluate systems at scale.
  • Error analysis.
    • Models latch onto spurious cues. Examples include mentions of medical issues for Article 3 and of higher-court appeals for Article 5.
    • No demographic bias was found on manual inspection.
    • The authors describe this as the first rationale-extraction work to fully fine-tune pre-trained Transformers end-to-end.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Law

Cited by

93 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 91 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.22

@inproceedings{chalkidis2021paragraph,
  title = {{Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases}},
  author = {Chalkidis, Ilias and Fergadiotis, Manos and Tsarapatsanis, Dimitrios and Aletras, Nikolaos and Androutsopoulos, Ion and Malakasiotis, Prodromos},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.22},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.22}
}

References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preoţiuc-Pietro, and Vasileios Lampos. 2016. Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective. PeerJ Computer Science, 2:e93. find
  2. David Alvarez-Melis and Tommi Jaakkola. 2017. A causal framework for explaining the predictions of black-box sequence-to-sequence models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 412– 421, Copenhagen, Denmark. Association for Computational Linguistics. find
  3. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias: There’s software used across the country to predict future criminals. and it’s biased against blacks. ProPublica. find
  4. Woodrow Barfield. 2020. The Cambridge Handbook of the Law of Algorithms. Cambridge Law Handbooks. Cambridge University Press. find
  5. Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963–2977, Florence, Italy. Association for Computational Linguistics. Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computational Linguistics. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT). find
  7. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443–4458, Online. Association for Computational Linguistics. find
  8. Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(10). find
  9. Madeleine Clare Elish, William Isaac, and Richard Zemel, editors. 2021. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAccT 2021). Association for Computing Machinery, Online. find
  10. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, page 2672–2680, Montreal, Canada. find
  11. Bryce Goodman and Seth Flaxman. 2017. European union regulations on algorithmic decision-making and a “right to explanation”. AI Magazine, 38(3):50–57. find
  12. David John Harris. 2018. Harris, O’Boyle, and War-brick : Law of the European Convention on Human Rights, 4rd edition. edition. Oxford University Press. find
  13. Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in english. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 78–87. find our card
  14. Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018. Few-shot charge prediction with discriminative legal attributes. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 487–498. find
  15. Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. Finding of EMNLP. find our card
  16. Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020. Reducing sentiment bias in language models via counterfactual evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 65–83, Online. Association for Computational Linguistics. find
  17. Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. 2019. A game theoretic approach to class-wise selective rationalization. In Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada. find
  18. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online. Association for Computational Linguistics. find
  19. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT). find our card
  20. Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. 2020. Learning to faithfully rationalize by construction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4459–4473, Online. Association for Computational Linguistics. find
  21. Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning to predict charges for criminal cases with legal basis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (Conference on Empirical Methods in Natural Language Processing (EMNLP)), pages 2727–2736. find
  22. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2009. Introduction to Information Retrieval. Cambridge University Press. find
  23. Daniel Martin Katz. 2012. Quantitative legal prediction-or-how I learned to stop worrying and start preparing for the data-driven future of the legal services industry. Emory Law Journal, 62:909. find
  24. Masha Medvedeva, Michel Vols, and Martijn Wieling. 2018. Judicial decisions of the European Court of Human Rights: Looking into the crystal ball. In Proceedings of the Conference on Empirical Legal Studies. find
  25. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL). find our card
  26. W. James Murdoch, Peter J. Liu, and Bin Yu. 2018. Beyond word importance: Contextual decomposition to extract interactions from lstms. In Proceedings of International Conference on Learning Representations (ICLR). find
  27. Diederik P. Kingma and Jim Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR). find
  28. Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should I trust you?": Explaining the predictions of any classifier. CoRR, abs/1602.04938. ↗ our card
  29. Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. In 31th Int. Conf. on Neural Information Processing Systems. find
  30. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365–4374, Hong Kong, China. Association for Computational Linguistics. find our card
  31. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107–117, Austin, Texas. find
  32. Zachary C. Lipton. 2018. The mythos of model interpretability. Commun. ACM, 61(10):36–43. find
  33. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how bert works. arXiv preprint arXiv:2002.12327. ↗ our card
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS). find
  35. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20, Hong Kong, China. Association for Computational Linguistics. find our card
  36. Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256. find
  37. Daniel Locke and Guido Zuccon. 2018. A test collection for evaluating legal case law search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 1261–1264, New York, NY, USA. Association for Computing Machinery. find
  38. Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. 2020. Beyond 512 tokens: Siamese multi-depth transformer-based hierarchical encoder for long-form document matching. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, page 1725–1734, New York, NY, USA. Association for Computing Machinery. find
  39. Christos Louizos and Max Welling. 2017. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of the International Conference on Machine Learning (ICML 2017), Sydney, Australia. find
  40. Zichao Yang et al. 2016. Hierarchical attention networks for document classification. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 1480–1489. find our card
  41. Mo Yu, Shiyu Chang, Yang Zhang, and Tommi Jaakkola. 2019. Rethinking cooperative rationalization: Introspective extraction and complement control. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4094–4103, Hong Kong, China. Association for Computational Linguistics. find
  42. Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using “annotator rationales” to improve machine learning for text categorization. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 260–267, Rochester, New York. Association for Computational Linguistics. find
  43. Ye Zhang, Iain Marshall, and Byron C. Wallace. 2016. Rationale-augmented convolutional neural networks for text classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 795–804, Austin, Texas. find
  44. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5218–5230, Online. Association for Computational Linguistics. find our card
  45. Haoxi Zhong, Guo Zhipeng, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal judgment prediction via topological learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3540–3549. find our card