Elamur Research⌕
☰
home / papers / 2022 / Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset

Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset

Paper · 2022 · Peter Henderson, Mark S. Krass, Lucia Zheng et al.

Summary

Teams that pretrain large language models remove private or toxic text in inconsistent, improvised ways and usually ignore context. The authors propose using legal practice as a guide, because courts and government agencies have long weighed openness against privacy and offensiveness. They release Pile of Law, an English corpus of about 256GB drawn from 35 public, mostly permissively licensed sources: court opinions and filings, contracts, statutes, regulations, agency documents, casebooks and legislative records. The paper compares how several jurisdictions redact personal details and turns these rules into practical guidance for dataset builders. It argues that the corpus carries implicit filtering norms that models could learn. Case studies on U.S. Supreme Court opinions show that standard toxicity classifiers often disagree, can misjudge historical or quoted language, and give very different scores depending on span length. The resource matters for two reasons. It is an openly licensed base for legal-domain pretraining, which could support access-to-justice tools. It is also a testbed for building filters that take context into account and reflect explicit values.

Key points

  • Pile of Law: about 256GB (and growing), 35 sources, hosted on Hugging Face (pile-of-law/pile-of-law). It includes existing collections such as EuroParl as subsets and focuses mainly on the U.S. legal system.
  • A survey of CCNet, C4, GPT-3, Gopher and The Pile finds that none explicitly filters potentially sensitive information (PSI). These efforts relied instead on deduplication, quality heuristics or ad hoc toxicity measures.
  • Courts in the U.S., Germany, China and Canada and U.S. administrative agencies share some baseline redactions: minors' identities, financial account numbers, birth dates and ID numbers. They differ on litigant names and criminal records.
  • The authors compared four toxicity tools on Supreme Court text: profanity-check, Perspective, Detoxify and Toxigen.
    • Sentence-level agreement between them (Cohen's κ) is low.
    • Dred Scott is not confidently flagged.
    • Civil rights rulings such as Loving and Hunter, which quote racist laws in order to strike them down, are often flagged.
  • Taking the top 5k Toxigen-flagged sentences and adding two neighboring sentences on each side lowered toxicity scores by roughly 55–57% (absolute).
  • A BERT-large-sized model pretrained on the corpus performed comparably to smaller, narrowly in-domain legal models.

Interesting ideas

  • I1. Use institutional rules as a starting point for data cleaning. Long-debated legal redaction practice, such as always removing ID numbers and minors' names, gives a defensible minimum standard. This avoids inventing filtering criteria from scratch, and the approach could carry over to corpora in other languages and countries, including Ukrainian ones.
  • I2. Learn filters from documents that were already redacted. Public records have been sanitized according to legal standards, so a model could learn context-dependent redaction decisions from them. For example, it could learn when a name is harmless (an attorney or an official) and when it is sensitive (asylum or juvenile cases).
  • I3. Measure agreement between filters before trusting any one of them. Comparing several toxicity classifiers with Cohen's κ, and tracking how agreement changes across time periods, exposes hidden value judgments and domain drift. It is a cheap check for any automatic filtering pipeline.
  • I4. Test how sensitive a filter is to context length. Re-scoring a flagged sentence together with its neighbors shows how much the verdict depends on span length. This matters when filtering long documents, where whole-document decisions might wrongly drop valuable texts or keep harmful ones.
  • I5. Check that quoting is not confused with endorsing. Cases that quote offensive material in order to reject it make good adversarial examples for evaluating filters, especially for historical, legal or journalistic corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 worksPolitical Science and International Relations · 206 works

🧭 Topics 1Research subjects and languages the work is about.Legal AI · 150 works

Cite

Cite as Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky and Daniel E. Ho. 2022. Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset. arXiv:2207.00220.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2207.00220 · arXiv 2207.00220

@misc{henderson2022pile,
  title = {{Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset}},
  author = {Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky and Daniel E. Ho},
  year = {2022},
  doi = {10.48550/arxiv.2207.00220},
  eprint = {2207.00220},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2207.00220}
}

References

127 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the Limits of Large Scale Pre-training. In International Conference on Learning Representations, 2021. find
  2. Allison Adams, Eric Aili, Daniel Aioanei, Rebecca Jonsson, Lina Mickelsson, Dagmar Mikmekova, Fred Roberts, Javier Fernandez Valencia, and Roger Wechsler. AnonyMate: A toolkit for anonymizing unstructured chat data. In Proceedings of the Workshop on NLP and Pseudonymisation, pages 1–7, 2019. find
  3. American Bar Association. Model Code of Judicial Conduct, 2007. https://www.americanbar.org/groups/professional_responsibility/publications/model_code_of_judicial_conduct. (1) (2) (3) (4) 0.05 0.00 0.81 0.54 0.60 0.94 0.50 0.26 0.54 ↗
  4. Noman Ashraf, Arkaitz Zubiaga, and Alexander Gelbukh. Abusive language detection in youtube comments leveraging replies as conversational context. PeerJ Computer Science, 7: e742, 2021. find
  5. Tuomas Aura, Thomas A Kuhn, and Michael Roe. Scanning electronic documents for personally identifiable information. In Proceedings of the 5th ACM workshop on Privacy in Electronic Society, pages 41–50, 2006. find
  6. Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 15479–15488, 2019. find
  7. Azeem Bande-Ali and Walker Boyle. U.S. Supreme Court annotated transcripts. Github, 2019. https://github.com/walkerdb/supreme_court_transcripts. ↗
  8. Alexander Baturo, Niheer Dasandi, and Slava J Mikhaylov. Understanding state preferences with text as data: Introducing the UN General Debate corpus. Research & Politics, 4(2), 2017. find
  9. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The Pushshift Reddit Dataset. In 14th International Conference on Web and Social Media, pages 830–839, 2020. find
  10. Steven Bellman, Eric J. Johnson, Stephen J. Kobrin, and Gerald L. Lohse. International Differences in Information Privacy Concerns: A Global Survey of Consumers. The Information Society, 20(5):313–324, 2004. find
  11. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. find
  12. Janine Benedet. Judicial Misconduct in the Sexual Assault Trial. U.B.C. L. Rev., 52:1, 2019. find
  13. Mary Frances Berry. The Pig Farmer’s Daughter and Other Tales of American Justice: Episodes of racism and sexism in the courts from 1865 to the present. Vintage, 2011. find
  14. Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. Tax Law NLP Resources, 2020. https://doi.org/10.7281/T1/N1X6I4. ↗
  15. Michael J Bommarito II, Daniel Martin Katz, and Eric M Detterman. LexNLP: Natural language processing and information extraction for legal and regulatory texts. In Research Handbook on Big Data Law. Edward Elgar Publishing, 2021. find
  16. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the Opportunities and Risks of Foundation Models. arXiv e-prints, art. arXiv:2108.07258, August 2021. ↗
  17. Łukasz Borchmann, Dawid Wisniewski, Andrzej Gretkowski, Izabela Kosmala, Dawid Jurkiewicz, Łukasz Szałkiewicz, Gabriela Pałka, Karol Kaczmarek, Agnieszka Kaliska, and Filip Graliński. Contract Discovery: Dataset and a Few-Shot Semantic Retrieval Challenge with Competitive Baselines. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4254–4268. Association for Computational Linguistics, November 2020. find
  18. Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1664–1674, Hong Kong, China, November 2019. Association for Computational Linguistics.
  19. British Columbia Law Society. Code of Professional Conduct for British Columbia, 2021. https://www.lawsociety.bc.ca/support-and-resources-for-lawyers/act-rules-and-code/code-of-professional-conduct-for-british-columbia/. ↗
  20. Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What Does it Mean for a Language Model to Preserve Privacy? In ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22), 2022. find
  21. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 1877–1901, 2020.
  22. Dana Burchardt. Backlash against the Court of Justice of the EU? The Recent Jurisprudence of the German Constitutional Court on EU Fundamental Rights as a Standard of Review. German Law Journal, 21:1–18, 2020. doi: doi:10.1017/glj.2020.16. ↗
  23. Naomi R Cahn. Looseness of legal language: The reasonable woman standard in theory and in practice. Cornell L. Rev., 77:1398, 1991. find
  24. Canadian Judicial Council. Ethical Principles for Judges, 2004. https://cjc-ccm.ca/cmslib/general/news_pub_judicialconduct_Principles_en.pdf. ↗
  25. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284, 2019.
  26. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021.
  27. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. ↗
  28. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. Large-scale multi-label text classification on EU legislation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6314–6322, Florence, Italy, 2019. Association for Computational Linguistics.
  29. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. LEGAL-BERT: The Muppets straight out of Law School. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904, 2020.
  30. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. Lexglue: A benchmark dataset for legal language understanding in english. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, 2022.
  31. Sumi Cho. Post-racialism. Iowa L. Rev., 94:1589, 2008. find
  32. Maximin Coavoux, Shashi Narayan, and Shay B Cohen. Privacy-preserving Neural Representations of Text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1–10, 2018. find
  33. I Glenn Cohen and Michelle M Mello. HIPAA and protecting health information in the 21st century. JAMA, 320(3):231–232, 2018. find
  34. Elaine Craig. The ethical obligations of defence counsel in sexual assault cases. Osgoode Hall L. J., 51:427, 2013. find
  35. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120–128, 2019. find
  36. Department of Veterans Affairs. Health Insurance Portability and Accountability Act Applicability in VBA, 2003. https://www.va.gov/ogc/docs/ADV3-2003.pdf. ↗
  37. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.
  38. Jana DiCosmo. Racism in the Legal Profession: A Racist Lawyer Is an Incompetent Lawyer. Nat’l Law. Guild Rev., 75:82, 2018. find
  39. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73, 2018. find
  40. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286–1305, 2021. find
  41. Zachary Elkins, Tom Ginsburg, James Melton, Robert Shaffer, Juan F Sequeda, and Daniel P Miranker. Constitute: The world’s constitutions to read, search, and compare. Journal of web semantics, 27:10–18, 2014. find
  42. Oluwaseyi Feyisetan, Abhinav Aggarwal, Zekun Xu, and Nathanael Teissier. Research challenges in designing differentially private text generation mechanisms. arXiv preprint arXiv:2012.05403, 2020. ↗
  43. Richard Thompson Ford. Racial epithets and racial etiquette. Capital University Law Review, 49(4):527–534, 2021. find
  44. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. ↗
  45. Alan E Garfield. To swear or not to swear: using foul language during a Supreme Court oral argument. Wash. U. L. Rev., 90:279, 2012. find
  46. Jamal Greene. The anticanon. Harv. L. Rev., 125:379, 2011. find
  47. Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N Asokan. All You Need is “Love” Evading Hate Speech Detection. In Proceedings of the 11th ACM workshop on artificial intelligence and security, pages 2–12, 2018. find
  48. Suchin Gururangan, Dallas Card, Sarah K Drier, Emily K Gade, Leroy Z Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A Smith. Whose language counts as high quality? Measuring language ideologies in text data selection. arXiv preprint arXiv:2201.10474, 2022. ↗
  49. Laura Hanu and Unitary team. Detoxify. Github, 2020. https://github.com/unitaryai/detoxify. ↗
  50. Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604, 2018. ↗
  51. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509, 2022. ↗
  52. David Hausman, Daniel E Ho, Mark Krass, and Anne M McDonough. Executive Control of Agency Adjudication: Capacity, Selection and Precedential Rulemaking. Journal of Law, Economics & Organization, 40, 2024. find
  53. Allison Hegel, Marina Shah, Genevieve Peaslee, Brendan Roof, and Emad Elwany. The Law of Large Documents: Understanding the Structure of Legal Contracts Using Visual Cues. arXiv preprint arXiv:2107.08128, 2021. ↗
  54. Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21(248):1–43, 2020. find
  55. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. arXiv preprint arXiv:2103.06268, 2021. ↗
  56. Zihan Huang, Charles Low, Mengqiu Teng, Hongyi Zhang, Daniel E Ho, Mark S Krass, and Matthias Grabmair. Context-aware legal citation recommendation using deep learning. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pages 79–88, 2021. find
  57. Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerda, Javier Jorge, Nahuel Roselló, Adria Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8229–8233. IEEE, 2020. find
  58. Yacine Jernite, Huu Nguyen, Stella Biderman, Anna Rogers, Maraim Masoud, Valentin Danchev, Samson Tan, Alexandra Sasha Luccioni, Nishant Subramani, Isaac Johnson, Somaieh Nikpoor, Jörg Frohberg, Aaron Gokaslan, Peter Henderson, Rishi Bommasani, and Margaret Mitchell. Data governance in the age of large-scale data-driven language technology. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 2206–2222, 2022. find
  59. Jigsaw and Google’s Counter Abuse Technology team. Perspective, 2021. https://perspectiveapi.com/. ↗
  60. Sheri Lynn Johnson, John H Blume, and Patrick M Wilson. Racial Epithets in the Criminal Process. Mich. St. L. Rev., page 755, 2011. find
  61. Lisa LaPlant Jon Quandt, Eric Mill. govinfo. Github. https://github.com/usgpo/api, 2018. ↗
  62. Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.06539, 2022. ↗
  63. Randall Kennedy and Eugene Volokh. The new taboo: Quoting epithets in the classroom and beyond. Cap. U. L. Rev., 49:1, 2021. find
  64. Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit, pages 79–86, 2005. find
  65. Satyapriya Krishna, Rahul Gupta, and Christophe Dupuy. ADePT: Auto-encoder based Differentially Private Text Transformation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2435–2439, 2021. find
  66. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700, 2019. ↗
  67. David S Law and Mila Versteeg. The declining influence of the United States Constitution. N. Y. U. L. Rev., 87:762, 2012. find
  68. Law Society of Ontario. Rules of Professional Conduct, 2022. https://lso.ca/about-lso/legislation-rules/rules-of-professional-conduct. ↗
  69. Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. Do language models plagiarize? arXiv preprint arXiv:2203.07618, 2022. ↗
  70. Alyssa Lees, Daniel Borkan, Ian Kivlichan, Jorge Nario, and Tesh Goyal. Capturing covertly toxic speech via crowdsourcing. In Proceedings of the First Workshop on Bridging Human– Computer Interaction and Natural Language Processing, pages 14–20, Online, April 2021. Association for Computational Linguistics. find
  71. Spyretta Leivaditi, Julien Rossi, and Evangelos Kanoulas. A benchmark for lease contract review. arXiv preprint arXiv:2010.10386, 2020. ↗
  72. Mark A Lemley and Bryan Casey. Fair learning. Tex. L. Rev., 99:743, 2020. find
  73. Justin D Levinson, Mark W Bennett, and Koichi Hioki. Judging implicit bias: A national empirical study of judicial sterotypes. Fla. L. Rev., 69:63, 2017. find
  74. Yitong Li, Timothy Baldwin, and Trevor Cohn. Towards robust and privacy-preserving text representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 25–30, 2018.
  75. Benjamin L Liebman, Margaret E Roberts, Rachel E Stern, and Alice Z Wang. Mass Digitization of Chinese Court Decisions. Journal of Law and Courts, Fall:176–201, 2020. find
  76. Marco Lippi, Przemysław Pałka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni. Claudette: an automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law, 27(2):117–139, 2019.
  77. Michael Lissner and Brian W Carver. Courtlistener. com: A platform for researching and staying abreast of the latest law. 2010. http://www.courtlistener.com. ↗
  78. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. ↗
  79. Eneldo Loza Mencía and Johannes Fürnkranz. Efficient multilabel classification algorithms for large-scale problems in the legal domain. In Semantic Processing of Legal Texts, pages 192–215. Springer, 2010. find
  80. Lingjuan Lyu, Xuanli He, and Yitong Li. Differentially private representation for nlp: Formal guarantee and an empirical study on privacy and fairness. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2355–2365, 2020. find
  81. Sibella Matthews, Vincent Schiraldi, and Lael Chester. Youth justice in Europe: Experience of Germany, the Netherlands, and Croatia in providing developmentally appropriate responses to emerging adults in the criminal justice system. Justice Evaluation Journal, 1(1):59–81, 2018. find
  82. H Brendan McMahan, Galen Andrew, Ulfar Erlingsson, Steve Chien, Ilya Mironov, Nicolas Papernot, and Peter Kairouz. A general approach to adding differential privacy to iterative training procedures. arXiv preprint arXiv:1812.06210, 2018. ↗
  83. Eric Mill. Opening up government reports through teamwork and open data. OpenGov Voices, 2014. Github., 2022. https://github.com/microsoft/ ↗
  84. Suzanne J Miller. Judicial language in new jersey sexual violence cases. Rutgers U. L. Rev., 73:141, 2020. find
  85. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019. find
  86. Nancy Morawetz. A better balance for federal rules governing public access to appeal records in immigration cases. Hastings LJ, 69:1271, 2017. find
  87. Deborah A Morgan. Not gay enough for the government: Racial and sexual stereotypes in sexual orientation asylum cases. Law & Sexuality: Rev. Lesbian, Gay, Bisexual & Transgender Legal Issues, 15:135, 2006. find
  88. Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. Abusive language detection in online user content. In Proceedings of the 25th international conference on world wide web, pages 145–153, 2016. find
  89. Beau Norgeot, Kathleen Muenzen, Thomas A Peterson, Xuancheng Fan, Benjamin S Glicksberg, Gundolf Schenk, Eugenia Rutenberg, Boris Oskotsky, Marina Sirota, Jinoos Yazdany, et al. Protected health information filter (philter): accurately and securely de-identifying free-text clinical notes. NPJ digital medicine, 3(1):1–8, 2020. find
  90. Nicholas M Pace and Julia Pollak. Provider fraud in california workers’ compensation: Selected issues, 2017. https://www.rand.org/content/dam/rand/pubs/research_reports/RR1700/RR1703/RAND_RR1703.pdf. ↗
  91. Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479–3495, Seattle, United States, July 2022. Association for Computational Linguistics.
  92. Prakash Poudyal, Jaromír Šavelka, Aagje Ieven, Marie Francine Moens, Teresa Gonçalves, and Paulo Quaresma. Echr: legal corpus for argument mining. In Proceedings of the 7th Workshop on Argument Mining, pages 67–75, 2020. find
  93. Praatika Prasad. Implicit racial biases in prosecutorial summations: Proposing an integrated response. Fordham L. Rev., 86:3091, 2017. find
  94. Reid Pryzant, Kelly Shen, Dan Jurafsky, and Stefan Wagner. Deconfounded lexicon induction for interpretable social science. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1615–1625, 2018. find
  95. National Historical Publications and Records Commission. Founders online, 2010. https://founders.archives.gov/. ↗
  96. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. ↗
  97. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. ↗
  98. Abigail M Reecer. The ethical dilemmas of the office of legal counsel in the wake of a whistleblower complaint. Geo. J. Legal Ethics, 33:769, 2020. find
  99. Douglas Rice, Jesse H Rhodes, and Tatishe Nteta. Racial bias in legal language. Research & Politics, 6(2):2053168019848930, 2019. find
  100. Federico Ruggeri, Francesca Lagioia, Marco Lippi, and Paolo Torroni. Detecting and explaining unfairness in consumer contracts through memory networks. Artificial Intelligence and Law, pages 1–34, 2021. find
  101. Steven Ruggles, Catherine Fitch, Diana Magnuson, and Jonathan Schroeder. Differential privacy and census data: Implications for social and economic research. In AEA papers and proceedings, volume 109, pages 403–08, 2019. find
  102. Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. arXiv preprint arXiv:1910.14659, 2019. ↗
  103. Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–15, 2021. find
  104. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. ↗
  105. Arman Sarvarian. Common ethical standards for counsel before the european court of justice and european court of human rights. European Journal of International Law, 23(4):991–1014, 2012. find
  106. Eran Shalev. Ancient masks, american fathers: Classical pseudonyms during the american revolution and early republic. Journal of the Early Republic, 23(2):151–172, 2003. find
  107. Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 1310–1321, 2015. find
  108. Amy Sholsberg, Evan Mandery, and Valerie West. The expungement myth. Albany Law Review, 75:1229–1242, 2011. find
  109. Congzheng Song and Ananth Raghunathan. Information Leakage in Embedding Models, page 377–390. Association for Computing Machinery, New York, NY, USA, 2020. ISBN 9781450370899. find
  110. Harold J Spaeth, Lee Epstein, Andrew D Martin, Jeffrey A Segal, Theodore J Ruger, and Sara C Benesh. Supreme court database, version 2013 release 01. Database at http://supremecourtdatabase.org, 2013. ↗
  111. Mike Spector, Jaimi Dowdell, and Benjamin Lesser. How secrecy in U.S. courts hobbles the regulators meant to protect the public. Reuters, 2010. https://www.reuters.com/investigates/special-report/usa-courts-secrecy-regulators. ↗
  112. Dan Sperber and Deirdre Wilson. Irony and the use-mention distinction. Philosophy, 3: 143–184, 1981. find
  113. Supreme Court of Canada. Sherman Estate v. Donovan, 2021. https://www.canlii.org/en/ca/scc/doc/2021/2021scc25/2021scc25.html. ↗
  114. Mark V Tushnet. The Warren Court in historical and political perspective. University of Virginia Press, 1993. Annual Report 2015–2016. find
  115. U.K. Judicial Conduct Investigations Office. 2016. https://jciodev.microsoftcrmportals.com/_entity/annotation/61785fbb-752a-eb11-a813-000d3a0bacd3. ↗
  116. United Kingdom Bar Standards Board. Bar Standards Board Handbook. https://www.barstandardsboard.org.uk/the-bsb-handbook.html?part=E3FF76D3-9538-4B97-94C02111664E5709&audience=&q=. ↗
  117. United Nations Office on Drugs and Crime. Commentary on the Bangalore Principles of Judicial Conduct. 2007. https://www.unodc.org/documents/nigeria/publications/Otherpublications/Commentry_on_the_Bangalore_principles_of_Judicial_Conduct.pdf. ↗
  118. United States Court of Appeals for the Second Circuit. Sealed Plaintiff v. Sealed Defendant, 2008. find
  119. U.S. Election Assistance Commission. Availability of state voter file and confidential information, 2020. https://www.eac.gov/sites/default/files/voters/Available_Voter_File_Information.pdf. ↗
  120. Betty van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. Challenges for toxic comment classification: An in-depth error analysis. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 33–42, 2018.
  121. Eugene Volokh. Pseudonymous litigation, 2021. https://www2.law.ucla.edu/volokh/pseudonym.pdf. ↗
  122. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003–4012, 2020.
  123. Wenjie Yin and Arkaitz Zubiaga. Towards generalisable hate speech detection: a review on obstacles and solutions. PeerJ Computer Science, 7:e598, 2021. find
  124. Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, and Sherman SM Chow. Differential privacy for text analytics via natural text sanitization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3853–3866, 2021. find
  125. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open Pre-trained Transformer Language Models. arXiv preprint arXiv:2205.01068, 2022. ↗
  126. Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. When does pretraining help? assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pages 159–168, 2021.
  127. Victor Zhou. profanity-check. Github. https://github.com/vzhou842/profanity-check, 2019. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/henderson-2022-pile-law-learning-responsible-data/