Elamur Research
☰
home / papers / 2024 / CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering

CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2404.04302

Summary

The paper tackles a weakness of LLM-based legal question answering: generated answers often lack grounding in verifiable evidence. The authors propose CBR-RAG, which uses the retrieval stage of Case-Based Reasoning (CBR), together with its indexing vocabulary and similarity knowledge, to pick past cases that are added to the LLM prompt as context. Each case holds a question, a supporting text snippet, extracted entities and an answer. The casebase comes from the Open Australian Legal Question-Answering (ALQA) dataset.

The study compares general and domain-specific embeddings (BERT, LegalBERT, AnglEBERT) under intra-attribute, inter-attribute and hybrid similarity. Answers are generated with Mistral-7B and scored by cosine similarity to reference answers. Every hybrid configuration beat a no-retrieval baseline. The strongest setup combined hybrid AnglEBERT, three neighbours and full cases as context.

The work matters because it shows that structured, case-based retrieval can measurably improve evidence-backed generation in an expert domain. It also finds that the choice of embedding for representing cases is crucial.

Key points

  • The casebase is ALQA: 2,124 GPT-4-synthesised QA pairs, each backed by a snippet from the Open Australian Legal Corpus. Of 785 distinct legal acts, 57% of cases cite none, so the authors index with neural embeddings rather than acts.
  • The test set has 32 manually reviewed QA pairs. Mistral generated them by merging pairs of cases that share legal acts.
  • Retrieval was evaluated as F1@k with kNN, testing prime values of k from 1 to 37. The best result came from hybrid AnglEBERT with weights [0.25, 0.40, 0.35] at k = 3.
  • In generation, the No-RAG cosine score was 0.8967. Hybrid AnglEBERT with k = 3 and full-case context reached 0.9141, roughly a 1.94% gain.
  • A one-tailed t-test showed the gain was significant at 95% over No-RAG and hybrid LegalBERT (k = 3), and at 90% over hybrid BERT (k = 1). Code is published at github.com/rgu-iit-bt/cbr-for-legal-rag.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning · Law

Cited by

37 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 36 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.04302 · arXiv 2404.04302

@misc{wiratunga2024cbr,
  title = {{CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering}},
  author = {Wiratunga, Nirmalie and Abeyratne, Ramitha and Jayawardena, Lasal and Martin, Kyle and Massie, Stewart and Nkisi-Orji, Ikechukwu and Weerasinghe, Ruvan and Liret, Anne and Fleisch, Bruno},
  year = {2024},
  doi = {10.48550/arxiv.2404.04302},
  eprint = {2404.04302},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.04302}
}

References

5 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Aleven, V., Ashley, K.D.: Teaching case-based argumentation through a model and examples: Empirical evaluation of an intelligent learning environment. In: Artificial intelligence in education. vol. 39, pp. 87–94. Citeseer (1997) 2. Asai, A., Wu, Z., Wang, Y., Sil, A., Hajishirzi, H.: Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511 (2023) 3. Ashley, K.D.: Reasoning with cases and hypotheticals in hypo. International journal of man-machine studies 34(6), 753–796 (1991) 4. Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., Shah, R.: Signature verification using a” siamese” time delay neural network. Advances in neural information processing systems 6 (1993) 5. Brüninghaus, S., Ashley, K.D.: The role of information extraction for textual cbr. In: International Conference on Case-Based Reasoning. pp. 74–89. ↗ our card
  2. Springer (2001) 6. Butler, U.: Open australian legal corpus (2024), https://huggingface.co/datasets/ umarbutler/open-australian-legal-corpus 7. Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., Androutsopoulos, I.: Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559 (2020) 8. Chalkidis, I., Jana, A., Hartung, D., Bommarito, M., Androutsopoulos, I., Katz, D.M., Aletras, N.: Lexglue: A benchmark dataset for legal language understanding in english (2022) 9. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) N. Wiratunga et al. 10. Guha, N., Nyarko, J., Ho, D.E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D.N., Zambrano, D., Talisman, D., Hoque, E., Surani, F., Fagan, F., Sarfaty, G., Dickinson, G.M., Porat, H., Hegland, J., Wu, J., Nudell, J., Niklaus, J., Nay, J., Choi, J.H., Tobia, K., Hagan, M., Ma, M., Livermore, M., Rasumov-Rahe, N., Holzenberger, N., Kolt, N., Henderson, P., Rehaag, S., Goel, S., Gao, S., Williams, S., Gandhi, S., Zur, T., Iyer, V., Li, Z.: Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models (2023) 11. Hacker, P., Engel, A., Mauer, M.: Regulating chatgpt and other large generative ai models. In: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. pp. 1112–1123 (2023) 12. Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2023) 13. Lai, J., Gan, W., Wu, J., Qi, Z., Yu, P.S.: LLMs in law: A survey (2023) 14. Lee, J.S.: Lexgpt 0.1: pre-trained gpt-j models with pile of law (2023) 15. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33, 9459–9474 (2020) 16. Li, X., Li, J.: Angle-optimized text embeddings (2023) 17. Rissland, E.L., Daniels, J.J.: A hybrid cbr-ir approach to legal information retrieval. In: Proceedings of the 5th international conference on Artificial intelligence and law. pp. 52–61 (1995) 18. Tang, C., Liu, Z., Ma, C., Wu, Z., Li, Y., Liu, W., Zhu, D., Li, Q., Li, X., Liu, T., Fan, L.: Policygpt: Automated analysis of privacy policies with large language models (2023) 19. Thulke, D., Daheim, N., Dugast, C., Ney, H.: Efficient retrieval augmented generation from unstructured knowledge for task-oriented dialog. arXiv preprint arXiv:2102.04643 (2021) 20. Tuggener, D., von Däniken, P., Peetz, T., Cieliebak, M.: LEDGAR: A large-scale multi-label corpus for text classification of legal provisions in contracts. In: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of the Twelfth Language Resources and Evaluation Conference. pp. 1235–1241. ↗ our card
  3. European Language Resources Association, Marseille, France (May 2020), https://aclanthology.org/2020.lrec-1.155 21. Tuggener, D., von Däniken, P., Peetz, T., Cieliebak, M.: LEDGAR: A large-scale multi-label corpus for text classification of legal provisions in contracts. In: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of the Twelfth Language Resources and Evaluation Conference. pp. 1235–1241. ↗
  4. European Language Resources Association, Marseille, France (May 2020), https://aclanthology.org/2020.lrec-1.155 22. Upadhyay, A., Massie, S.: A case-based approach for content planning in data-to-text generation. In: International Conference on Case-Based Reasoning. pp. 380– 394. ↗
  5. Springer (2022) 23. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017) CBR for RAG in LLMs for Legal QA 24. Wiratunga, N., Koychev, I., Massie, S.: Feature selection and generalisation for retrieval of textual cases. In: European Conference on Case-Based Reasoning. pp. 806–820. find