Elamur Research
☰
home / papers / 2024 / Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools

Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2405.20362

Summary

Legal technology vendors have marketed their AI research assistants, which are built on retrieval-augmented generation (RAG), as largely or fully free of fabricated content. Because these products are closed, no one had tested those claims independently. The authors ran the first pre-registered empirical evaluation of such commercial tools. They hand-built a set of more than 200 legal questions designed to probe weak spots. They submitted these questions to Lexis+ AI, Westlaw AI-Assisted Research, Ask Practical Law AI, and GPT-4 as a general-purpose baseline, and then manually graded every answer for correctness and faithfulness to the cited authority. The authors also proposed a typology for hallucinations. Under it, an answer counts as hallucinated if it states something false or wrongly claims that a source backs a proposition. RAG made answers better than GPT-4's, but the specialized tools still hallucinated in roughly one out of every six to three answers. The findings undercut vendor marketing and show that lawyers still need to check outputs. The authors argue for transparent public benchmarking of legal AI.

Key points

  • The three commercial tools hallucinated on about 17% to 33% of queries. This was fewer than GPT-4 but well above zero.
  • Lexis+ AI performed best, with about 65% of answers accurate. Westlaw AI-Assisted Research was accurate on about 42% of queries, but its hallucination rate was roughly double that of the remaining legal products.
  • Ask Practical Law AI gave incomplete responses, either refusals or answers lacking proper grounding, on more than 60% of queries. This was the highest rate among the systems tested.
  • The benchmark has over 200 pre-registered queries. The authors say the queries, tool outputs, and labels will be released once the paper is published.
  • The paper also flags risks for vendors that overstate reliability. These include liability for false advertising under the Lanham Act and possible tort claims.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning · Law

Cited by

105 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 104 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.20362 · arXiv 2405.20362

@misc{magesh2024hallucination,
  title = {{Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools}},
  author = {Magesh, Varun and Surani, Faiz and Dahl, Matthew and Suzgun, Mirac and Manning, Christopher D. and Ho, Daniel E.},
  year = {2024},
  doi = {10.48550/arxiv.2405.20362},
  eprint = {2405.20362},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.20362}
}

References

113 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ayush Agrawal, Mirac Suzgun, Lester Mackey, and Adam Tauman Kalai. 2023. Do Language Models Know When They’re Hallucinating References? arXiv preprint. find
  2. Bob Ambrogi. 2023. Major Thomson Reuters News: Westlaw Gets Generative AI Research Plus Integration with Casetext CoCounsel; Gen AI Coming Soon to Practical Law. find
  3. Bob Ambrogi. 2024. LawNext: Thomson Reuters’ AI Strategy for Legal, with Mike Dahn, Head of Westlaw, and Joel Hron, Head of AI. find
  4. American Bar Association. 2018. Alphabetical List of Jurisdictions Adopting Model Rules. find
  5. Olufunmilayo B. Arewa. 2006. Open Access in a Closed Universe: Lexis, Westlaw, Law Schools, and the Legal Information Market. Lewis & Clark Law Review, 10(4):797–840. find
  6. Michael Atleson. 2023. Keep your AI claims in check. find
  7. Joseph J. Avery, Patricia Sánchez Abril, and Alissa del Riego. 2023. ChatGPT, Esq.: Recasting Unauthorized Practice of Law in the Era of Generative AI. Yale Journal of Law & Technology, 26(1):64–129. find
  8. BARBRI, Inc. 2013. Multistate Testing Practice Questions. find
  9. Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024. Seven Failure Points When Engineering a Retrieval Augmented Generation System. arXiv preprint. find
  10. Deborah Beim and Kelly Rader. 2019. Legal Uniformity in American Courts. Journal of Empirical Legal Studies, 16(3):448–478. find
  11. Nicholas J Belkin. 2008. Some (what) grand challenges for information retrieval. In ACM SIGIR Forum, volume 42, pages 47–54. ACM New York, NY, USA. find
  12. Berkeley Law School. 2024. Generative AI Resources for Berkeley Law Faculty & Staff. find
  13. BIG-bench Authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. find our card
  14. Ryan C. Black and James F. Spriggs, II. 2013. The Citation and Depreciation of U.S. Supreme Court Precedent. Journal of Empirical Legal Studies, 10(2):325–358. find
  15. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2022. On the Opportunities and Risks of Foundation Models. arXiv preprint. find our card
  16. Nina Brown. 2023. Bots Behaving Badly: A Products Liability Approach to Chatbot-Generated Defamation. Journal of Free Speech Law, 3(2):389–424. find
  17. Ellie Campbell. 2024. Resources for Exploring the Benefits and Drawbacks of AI. find
  18. Casetext. 2023. GPT-4 alone is not a reliable legal solution—but it does enable one: CoCounsel harnesses GPT-4’s power to deliver results that legal professionals can rely on. find
  19. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Benchmarking large language models in retrieval-augmented generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(1616):17754–17762. find our card
  20. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. How is chatgpt’s behavior changing over time? arXiv preprint arXiv:2307.09009. ↗
  21. Colleen V. Chien and Miriam Kim. 2024. Generative AI and Legal Aid: Results from a Field Study and 100 Use Cases to Bridge the Access to Justice Gap. Loyola of Los Angeles Law Review, forthcoming. find
  22. Colleen V. Chien, Miriam Kim, Raj Akhil, and Rohit Rathish. 2024. How Generative AI Can Help Address the Access to Justice Gap Through the Courts. Loyola of Los Angeles Law Review, forthcoming. find
  23. Jonathan H. Choi, Amy Monahan, and Daniel Schwarcz. 2024. Lawyering in the Age of Artificial Intelligence. find
  24. Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37–46. ↗
  25. Jack Collens, Rachel Reimer, Gerald Schifman, and Pamela Wilkinson. 2024. AI Survey: Where Artificial Intelligence Stands in the Legal Industry. find
  26. Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The Power of Noise: Redefining Retrieval for RAG Systems. arXiv preprint. find
  27. Amy B. Cyphert. 2021. A Human Being Wrote This Law Review Article: GPT-3 and the Practice of Law. UC Davis Law Review, 55(1):401–444. find
  28. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. 2024. Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analaysis, forthcoming. find our card
  29. Fabrizio Dell’Acqua, Edward McFowland, Ethan R. Mollick, Hila Lifshitz-Assaf, Katherine Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R. Lakhani. 2023. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. find
  30. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2023. Investigating data contamination in modern benchmarks for large language models. arXiv preprint arXiv:2311.09783. ↗ our card
  31. Mihailis E. Diamantis. 2023. Vicarious Liability for AI. Indiana Law Journal, 99(1):317–334. find
  32. Ronald Dworkin. 1986. Law’s Empire. Harvard University Press, Cambridge, MA. find
  33. James H. Fowler, Timothy R. Johnson, James F. Spriggs, II, Sangick Jeon, and Paul J. Wahlbeck. 2007. Network Analysis and the Law: Measuring the Legal Importance of Precedents at the U.S. Supreme Court. Political Analysis, 15(3):324–346. find
  34. Free Law Project. 2024. Courtlistener. find
  35. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint. find our card
  36. K. Goddard, A. Roudsari, and J. C. Wyatt. 2012. Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association: JAMIA, 19(1):121–127. find
  37. Google. 2024. Google search results for "practical law". Accessed: 2024-05-22. find
  38. Isabel Gottlieb. 2024. Paul Weiss Assessing Value of AI, But Not Yet on Bottom Line. Bloomberg Law. find
  39. Stuart Greenhill. 2024. Lawyers Cross into the New Era of Generative AI. find
  40. Gurbir S. Grewal. 2024. Remarks at Program on Corporate Compliance and Enforcement Spring Conference 2024. find
  41. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. 2023. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. arXiv preprint. find our card
  42. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of ICML’20, pages 3929–3938. JMLR.org. find
  43. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying Human and Statistical Evaluation for Natural Language Generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1689–1701, Minneapolis, Minnesota. Association for Computational Linguistics. find
  44. Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. 2022. Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset. arXiv preprint. find
  45. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations. find our card
  46. Justin Henry. 2024. We Asked Every Am Law 100 Law Firm How They’re Using Gen AI. Here’s What We Learned. The American Lawyer. find
  47. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12):248:1–248:38. find our card
  48. Adam Tauman Kalai and Santosh S. Vempala. 2023. Calibrated Language Models Must Hallucinate. arXiv preprint. find
  49. Sayash Kapoor, Peter Henderson, and Arvind Narayanan. 2024. Promises and Pitfalls of Artificial Intelligence for Legal Applications. Journal of Cross-disciplinary Research in Computational Law, forthcoming. find
  50. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. Association for Computational Linguistics. find our card
  51. Darla Wynon Kite-Jackson. 2023. 2023 Artificial Intelligence (AI) TechReport. Technical report, American Bar Association. find
  52. Law360. 2024. Tracking Federal Judge Orders On Artificial Intelligence. find
  53. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc. find
  54. LexisNexis. 2023. General terms and conditions. find
  55. LexisNexis. 2023. LexisNexis Launches Lexis+ AI, a Generative AI Solution with Linked Hallucination-Free Legal Citations. find
  56. LexisNexis. 2024. LexisNexis Launches Second-Generation Legal AI Assistant on Lexis+ AI. find
  57. Changmao Li and Jeffrey Flanigan. 2024. Task contamination: Language models may not be few-shot anymore. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18471–18480. find
  58. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023. Holistic Evaluation of Language Models. find
  59. Michael Lissner. 2022. Important opinions on courtlistener are now summarized by the top experts — judges. find
  60. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122–2132, Austin, Texas. Association for Computational Linguistics. find
  61. Megan Ma, Aparna Sinha, Ankit Tandon, and Jennifer Richards. 2024. find
  62. Generative AI Legal Landscape 2024. Technical report. find
  63. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. arXiv preprint. find
  64. Alva Markelius, Connor Wright, Joahna Kuiper, Natalie Delille, and Yu-Ting Kuo. 2024. The Mechanisms of AI Hype and Its Planetary and Social Costs. AI and Ethics. find
  65. Eric Martínez. 2024. Re-evaluating GPT-4’s bar exam performance. Artificial Intelligence and Law, pages 1–24. find
  66. Paul McGreel. 2024. I asked Lexas+ AI [sic] a simple question: "What cases have applied Students for Fair Admissions, Inc. v. Harvard College to the use of race in government decisionmaking?" This screenshot has the answer I received. Here are some of the (serious) problems with this answer. Twitter. find
  67. Timothy R. McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N. Halgamuge. 2024. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. Preprint, arXiv:2402.09880. ↗
  68. Eliza Mik. 2024. Caveat Lector: Large Language Models in Legal Practice. find
  69. Roger J. Miner. 1989. Remarks: Clerks of Judge Luther A. Wilgarten, Jr. find
  70. Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. 2023. Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation. arXiv preprint. find
  71. Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2023. Proving Test Set Contamination for Black-Box Language Models. In The Twelfth International Conference on Learning Representations. find
  72. Andreas Östling, Holli Sargeant, Huiyuan Xie, Ludwig Bull, Alexander Terenin, Leif Jonsson, Måns Magnusson, and Felix Steffek. 2024. The Cambridge Law Corpus: A Dataset for Legal AI Research. arXiv preprint. find
  73. Andrew Perlman. 2023. The Implications of ChatGPT for Legal Services and Society. The Practice, (March/April). find
  74. Inioluwa Deborah Raji, Emily Denton, Emily M Bender, Alex Hanna, and Amandalynne Paullada. 2021. AI and the Everything in the Whole Wide World Benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). find
  75. Inioluwa Deborah Raji, I. Elizabeth Kumar, Aaron Horowitz, and Andrew Selbst. 2022. The fallacy of ai functionality. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 959–972, New York, NY, USA. Association for Computing Machinery. find
  76. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv preprint. find our card
  77. Christopher C. Rivers. 2024. Moffatt v. Air Canada. find
  78. John G. Roberts. 2023. 2023 Year-End Report on the Federal Judiciary. Technical report. find
  79. Peter Salib. 2024. AI Outputs Are Not Protected Speech. find
  80. Daniel Schwarcz and Jonathan H. Choi. 2023. AI Tools for Lawyers: A Practical Guide. Minnesota Law Review Headnotes, 108:1–39. find
  81. Securities and Exchange Commission. 2024. Sec Charges Two Investment Advisers with Making False and Misleading Statements About Their Use of Artificial Intelligence. find
  82. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2023. Towards understanding sycophancy in language models. find our card
  83. Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. 2023. Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering. Transactions of the Association for Computational Linguistics, 11:1–17. find
  84. Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022. Human Evaluation of Conversations is an Open Problem: Comparing the sensitivity of various methods for evaluating dialogue agents. In Proceedings of the 4th Workshop on NLP for Conversational AI, pages 77–97, Dublin, Ireland. Association for Computational Linguistics. find
  85. Suffolk University. 2023. Law Faculty Guide to Artificial Intelligence: Practical Law AI (Westlaw). find
  86. Faiz Surani, Matthew Dahl, and Varun Magesh. 2024. Legal RAG hallucinations. find
  87. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada. Association for Computational Linguistics. find our card
  88. Jinzhe Tan, Hannes Westermann, and Karim Benyekhlef. 2023. ChatGPT as an Artificial Lawyer? In Proceedings of the ICAIL 2023 Workshop on Artificial Intelligence for Access to Justice, Braga, Portugal. CEUR Workshop Proceedings. find
  89. Task Force on Artificial Intelligence. 2024. Report and Recommendations of the New York State Bar Association Task Force on Artificial Intelligence. Technical report, New York State Bar Association. find
  90. The Florida Bar. 2024. Florida Bar Ethics Opinion. Technical Report 24-1, The Florida Bar. find
  91. The Library Innovation Lab. 2024. Transitions for the Caselaw Access Project. find
  92. The State Bar of California. 2023. Practical Guidance for the Use of Generative Artificial Intelligence in the Practice of Law. Technical report, The State Bar of California. find
  93. Thomson Reuters. 2019. Westlaw tip of the week: Checking cases with keycite. find
  94. Thomson Reuters. 2023. Introducing AI-Assisted Research: Legal research meets generative AI. find
  95. Thomson Reuters. 2024a. Accelerate how you find answers so you can practice with confidence. find
  96. Thomson Reuters. 2024b. Introducing Ask Practical Law AI on Practical Law: Generative AI meets legal how-to. find
  97. University of Washington. 2024. Artificial Intelligence. find
  98. Matthew van der Merwe, Ketan Ramakrishnan, and Markus Anderljung. 2024. Tort Law and Frontier AI Governance. Technical report, Lawfare. find
  99. Eugene Volokh. 2023. Large Libel Models? Liability for AI Output. Journal of Free Speech Law, 3(2):489–558. find
  100. Jeremy Waldron. 2002. Is the Rule of Law an Essentially Contested Concept (in Florida)? Law and Philosophy, 21(2):137–164. find
  101. Ed Walters. 2019. The Model Rules of Autonomous Conduct: Ethical Responsibilities of Lawyers and Artificial Intelligence. Georgia State University Law Review, 35(4):1073–1092. find
  102. Benjamin Weiser. 2023. Here’s What Happens When Your Lawyer Uses ChatGPT. The New York Times. find
  103. Benjamin Weiser and Jonah E. Bromwich. 2023. Michael Cohen Used Artificial Intelligence in Feeding Lawyer Bogus Cases. The New York Times. find
  104. Serena Wellen. 2024a. How Lexis+ AI Delivers Hallucination-Free Linked Legal Citations. find
  105. Serena Wellen. 2024b. Tech Innovation with LLMs Producing More Secure and Reliable Gen AI Results. find
  106. Peter Wills. 2024. Care for Chatbots. UBC Law Review, 73(3). find
  107. Yale Law School. 2024. Lexis and Westlaw Generative AI Products. find
  108. Nicole Yamane. 2020. Artificial Intelligence in the Legal Field and the Indispensable Human Element Legal Ethics Demands. Georgetown Journal of Legal Ethics, 33(3):877–890. find
  109. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. arXiv preprint. find our card
  110. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Thirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  111. Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the eighteenth international conference on artificial intelligence and law, pages 159–168. find
  112. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. 2023a. Representation engineering: A top-down approach to ai transparency. Preprint, arXiv:2310.01405. ↗ our card
  113. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023b. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. ↗ our card