Elamur Research⌕
☰
home / papers / 2026 / bLLeQA: Benchmarking LLMs for Grounded Legal Question-Answering in French and Dutch

bLLeQA: Benchmarking LLMs for Grounded Legal Question-Answering in French and Dutch

Paper · 2026 · Nikolay Banar, Ehsan Lotfi, Jens Van Nooten et al.

Summary

Legal question-answering tools could make the law easier to reach for ordinary people. However, evaluation data for retrieval-augmented generation (RAG) in civil-law settings and smaller languages such as Dutch is rare. The authors extend the French LLeQA dataset into bLLeQA, a parallel French–Dutch benchmark built on Belgian legislation.

To build it, they:

  • aligned the official Dutch versions of the legal articles, reusing bBSARD and scraping further pairs from Justel;
  • machine-translated the remaining questions and the answers with GPT-5.0, then had a bilingual legal expert post-edit them;
  • had the expert check that each answer is actually supported by its cited articles.

They then benchmark many open-weight and proprietary LLMs zero-shot across the whole pipeline: retrieval, citation extraction, refusal and answer generation, under three context settings.

Open-weight models keep pace with proprietary ones in retrieval and citation, but trail them in generation quality. No model reliably notices when the supplied sources are incomplete. Even the strongest configuration gives a flawed answer about one time in five. This shows that grounded legal QA remains far from dependable.

Key points

  • bLLeQA contains 1,461 QA pairs per language, grounded in a corpus of 25,982 aligned Belgian legal articles. It was derived from LLeQA, which has 1,868 French questions and about 28k articles. The article alignment rate is 93%.
  • Expert grounding review gave the following outcomes:
    • 72% of samples were kept unchanged.
    • 6.3% were fixed by removing over-cited articles.
    • 21.7% were dropped for problems such as subject mismatch, overly general context, missing information, procedural vs substantive mismatch, or overly long context.
  • Translation post-editing touched 20.8% of reference answers but only 1.4% of questions. The typical errors were untranslated abbreviations (e.g. MENA) and Walloon institutions lacking their Flemish counterparts (e.g. FAMIWAL).
  • Generation results:
    • The best proprietary models (GPT-5.4, Claude-Sonnet-4.6, Gemini-3-Flash) reach a mean G-Eval correctness of 4.2 with about 20% flawed outputs.
    • The top open-weight models (GLM-5, Qwen3.5-397B-A17B, Kimi-K2-Thinking) peak at 3.9 for French and 3.7 for Dutch, with about 35% flawed outputs.
  • Refusal is weak across the board. The best refusal F1 is roughly 56 (GPT-5.4) and 52 (Kimi-K2.5). Some high-correctness models refuse poorly.

Interesting ideas

  • I1. Have a human verify that the gold evidence truly suffices to justify each answer, and record a typed reason whenever it does not (wrong subject, too vague, missing fact, wrong kind of rule).
  • I2. Measure refusal by feeding the system deliberately incomplete evidence and checking whether it declines to answer instead of guessing.
  • I3. Score faithfulness to the context separately from answer correctness, and plot both against citation recall; the paper finds that being grounded does not guarantee being right, and that some missing evidence matters less than other evidence.
  • I4. Make citation of the supporting sources an explicit, separately evaluated output, rather than relying only on the answer text.
  • I5. Translation post-editing showed that most errors involve acronyms and region-specific institutions that need a mapped counterpart rather than a literal rendering.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Our fable and school-test QA sets could tag each item with the world-graph facts that support it, plus a failure category when support is lacking, so that reader errors can be separated from dataset errors.
  • I2 (learn): This works as a natural negative control: delete a supporting sentence from a story and require the reader to answer "unknown" rather than produce a confident answer.
  • I3 (answer): Readers could report which world facts each answer relied on, letting us test whether faulty answers come from unused facts or from wrongly weighted ones.
  • I4 (answer): Each answer could list the tree or frame nodes it was derived from, giving a checkable evidence trail in line with Mova's explainability goal.
  • I5 (back): When translating through the meta-world, named institutions and abbreviations should be resolved as entities with locale-specific equivalents, not passed through as surface strings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AntwerpAntwerp · Belgium17 works · 3,490 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksInformation Systems · 1,730 worksPolitical Science and International Relations · 206 works

🧭 Topics 3Research subjects and languages the work is about.Question answering · 487 worksLegal AI · 150 worksFrench · 83 works

Cite

Cite as Nikolay Banar, Ehsan Lotfi, Jens Van Nooten, Marija Kliocaite and Walter Daelemans. 2026. bLLeQA: Benchmarking LLMs for Grounded Legal Question-Answering in French and Dutch. Proceedings of the 4th Workshop on Towards Knowledgeable Foundation Models (KnowFM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.knowfm-1.4

@inproceedings{banar2026blleqa,
  title = {{bLLeQA: Benchmarking LLMs for Grounded Legal Question-Answering in French and Dutch}},
  author = {Nikolay Banar, Ehsan Lotfi, Jens {Van Nooten}, Marija Kliocaite and Walter Daelemans},
  booktitle = {Proceedings of the 4th Workshop on Towards Knowledgeable Foundation Models (KnowFM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.knowfm-1.4},
  url = {https://doi.org/10.18653/v1/2026.knowfm-1.4}
}

References

81 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024b. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bangkok, Thailand. Association for Computational Linguistics.
  2. Pawitsapak Akarajaradwong, Pirat Pothavorn, Chompakorn Chaksangchaichot, Panuthep Tasawong, Thi-tiwat Nopparatbundit, Keerakiat Pratai, and Sarana Nutanong. 2025. NitiBench: Benchmarking LLM frameworks on Thai legal question answering capabilities. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 34304–34327, Suzhou, China. Association for Computational Linguistics. find
  3. Cristian-George Craciun, Răzvan-Alexandru Smădu, Dumitru-Clementin Cercel, and Mihaela-Claudia Cercel. 2025. GRAF: Graph retrieval augmented by facts for Romanian legal multi-choice question answering. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12708–12742, Vienna, Austria. Association for Computational Linguistics. find
  4. Anthropic. 2025. Claude haiku 4.5. https://www.anthropic.com/claude/haiku. Accessed: 2026-02-09. ↗
  5. Anthropic. 2026. Introducing claude sonnet 4.6. https://www.anthropic.com/news/ claude-sonnet-4-6. Accessed: 2026-03-27. ↗
  6. DeepMind. 2025a. Gemini 2.5 flash model card. https://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-2-5-Flash-Lite-Model-Card.pdf. Accessed: 2026-03-27. ↗
  7. Farid Ariai, Joel Mackenzie, and Gianluca Demartini. 2025. Natural language processing for the legal domain: A survey of tasks, datasets, models, and challenges. ACM Comput. Surv., 58(6). ↗
  8. Arian Askari, Zihui Yang, Zhaochun Ren, and Suzan Verberne. 2024. Answer retrieval in legal community question answering. In Advances in Information Retrieval: 46th European Conference on Information Retrieval, ECIR 2024, Glasgow, UK, March 24–28, 2024, Proceedings, Part III, page 477–485, Berlin, Heidelberg. Springer-Verlag. find
  9. DeepMind. 2025b. Gemini 3 flash model card. https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-Flash-Model-Card. pdf. Accessed: 2026-02-10. ↗
  10. DeepMind. 2025c. Gemini 3 pro model card. https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-Pro-Model-Card.pdf. Accessed: 2026-02-10. {Nigel J.} Balmer, Ash Patel, Alexy Buck, Catrina Denvir, and Pascoe Pleasence. 2010. Knowledge, Capacity and the Experience of Rights Problems. Public Legal Education Network: PLENet. ↗
  11. DeepMind. 2026. Gemini 3.1 flash-lite model card. https://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-1-Flash-Lite-Model-Card.pdf. Accessed: 2026-03-27. ↗
  12. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135–146. ↗
  13. DeepSeek-AI. 2025. Deepseek-v3.2: Pushing the frontier of open large language models. ↗
  14. Marius Büttner and Ivan Habernal. 2024. Answering legal questions from laymen in German civil law system. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2015–2027, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  15. Xingyi Duan, Baoxin Wang, Ziyue Wang, Wentao Ma, Yiming Cui, Dayong Wu, Shijin Wang, Ting Liu, Tianxiang Huo, Zhen Hu, Heng Wang, and Zhiyuan Liu. 2019. CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension, page 439–451. Springer International Publishing. find
  16. Andong Chen, Feng Yao, Xinyan Zhao, Yating Zhang, Changlong Sun, Yun Liu, and Weixing Shen. 2023. Equals: A real-world dataset for legal question answering via reading chinese laws. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL ’23, page 71–80, New York, NY, USA. Association for Computing Machinery. find
  17. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians, Malta. Association for Computational Linguistics.
  18. Faizan Faisal and Umair Yousaf. 2024. Legal-uqa: A low-resource urdu-english dataset for legal question answering. Preprint, arXiv:2410.13013. ↗
  19. Steve Han, Gilberto Titericz Junior, Tom Balough, and Wenfei Zhou. 2026. Judge’s verdict: A comprehensive analysis of LLM judge capability through human agreement. ↗
  20. Jean-Philippe Fauconnier. 2015. French word embeddings. Accessed: 2026-01-05. find
  21. Congqing He, Haichuan Hu, Yanli Li, Hao Zhang, and Quanjun Zhang. 2026. A survey of large language models for legal tasks: Progress, prospects and challenges. Computer Science Review, 60:100906. ↗
  22. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland. Association for Computational Linguistics.
  23. Abe Bohan Hou, Orion Weller, Guanghui Qin, Eugene Yang, Dawn Lawrie, Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2025. CLERC: A dataset for U. S. legal case retrieval and retrieval-augmented analysis generation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7898–7913, Albuquerque, New Mexico. Association for Computational Linguistics. find
  24. Yuanning Feng, Sinan Wang, Zhengxiang Cheng, Yao Wan, and Dongping Chen. 2025. Are we on the right way to assessing llm-as-a-judge? Preprint, arXiv:2512.16041. ↗
  25. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. ↗
  26. GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, and 168 others. 2026. Glm-5: from vibe coding to agentic engineering. Preprint, arXiv:2602.15763. ↗
  27. JinaAI. 2025. jinaai/jina-reranker-v2-base-multilingual. https://huggingface.co/jinaai/ Ac-jina-reranker-v2-base-multilingual. cessed: 2026-02-09. ↗
  28. Tuba Gokhan and Ted Briscoe. 2025. Grounded answers from multi-passage regulations: Learning-torank for regulatory RAG. In Proceedings of the Natural Legal Language Processing Workshop 2025, pages 135–146, Suzhou, China. Association for Computational Linguistics. find
  29. Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. 2025. Abstentionbench: Reasoning LLMs fail on unanswerable questions. In ICML 2025 Workshop on Reliable and Responsible Foundation Models. find
  30. Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. 2023. Making large language models a better foundation for dense retrieval. Preprint, arXiv:2312.15503. ↗
  31. Tuba Gokhan, Kexin Wang, Iryna Gurevych, and Ted Briscoe. 2025. Shared task RIRAG-2025: Regulatory information retrieval and answer generation. In Proceedings of the 1st Regulatory NLP Workshop (RegNLP 2025), pages 1–4, Abu Dhabi, UAE. Association for Computational Linguistics. find
  32. Haitao Li, Yunqiu Shao, Yueyue Wu, Qingyao Ai, Yixiao Ma, and Yiqun Liu. 2024. Lecardv2: A large-scale chinese legal case retrieval dataset. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 2251–2260, New York, NY, USA. Association for Computing Machinery. find
  33. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  34. Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Sooriyarachchi, Baptiste Bout, and 101 others. 2026. Ministral 3. Preprint, arXiv:2601.08584. ↗
  35. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21 others. 2023. Legalbench: a collaboratively built benchmark for measuring legal reasoning in large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
  36. Bulou Liu, Zhenhao Zhu, Qingyao Ai, Yiqun Liu, and Yueyue Wu. 2024. Ledqa: A chinese legal case document-based question answering dataset. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM ’24, page 5385–5389, New York, NY, USA. Association for Computing Machinery. find
  37. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics.
  38. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. Preprint, arXiv:1301.3781. ↗
  39. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26. ↗
  40. Ehsan Lotfi, Nikolay Banar, and Walter Daelemans. 2025a. BEIR-NL: Zero-shot information retrieval benchmark for the Dutch language. In Proceedings of the 18th Workshop on Building and Using Comparable Corpora (BUCC), pages 36–45, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  41. Mistral AI. 2025. Mistral 3-675b instruct-2512 model https://huggingface.co/mistralai/ Mistral-Large-3-675B-Instruct-2512. cessed: 2026-02-10. ↗
  42. Ehsan Lotfi, Nikolay Banar, Nerses Yuzbashyan, and Walter Daelemans. 2025b. Bilingual BSARD: Extending statutory article retrieval to Dutch. In Proceedings of the 1st Regulatory NLP Workshop (Reg-NLP 2025), pages 10–21, Abu Dhabi, UAE. Association for Computational Linguistics. large card. ↗
  43. Ac-Moonshot AI. 2026. Kimi k2.5: Ai that sees, codes, and works like an expert. https://www.kimi.com/ ai-models/kimi-k2-5. Accessed: 2026-03-27. ↗
  44. OpenAI. 2025. Gpt-5 mini model. https://platform.openai.com/docs/models/gpt-5-mini. Accessed: 2026-02-10. ↗
  45. Antoine Louis, Vageesh Kumar Saxena, Gijs van Dijck, and Gerasimos Spanakis. 2025. ColBERT-XM: A modular multi-vector representation model for zero-shot multilingual information retrieval. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4370–4383, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  46. OpenAI. 2025a. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925. ↗
  47. OpenAI. 2025b. Introducing gpt-5. https://openai.com/index/introducing-gpt-5/. Accessed: 2025-11-13. ↗
  48. Antoine Louis and Gerasimos Spanakis. 2022. A statutory article retrieval dataset in French. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6789–6803, Dublin, Ireland. Association for Computational Linguistics. find
  49. OpenAI. 2025c. Introducing gpt-5.4. https://openai.com/index/us-EN/introducing-gpt-5. 4/. Accessed: 2026-03-27. ↗
  50. Nicholas Pipitone and Ghita Houir Alami. 2024. Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain. Preprint, arXiv:2408.10343. ↗
  51. Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2024. Interpretable long-form legal question answering with retrieval-augmented large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20):22266–22275.
  52. QwenTeam. 2026. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id=qwen3. 5. Accessed: 2026-03-27. ↗
  53. Yixiao Ma, Yunqiu Shao, Yueyue Wu, Yiqun Liu, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2021. Lecard: A legal case retrieval dataset for chinese law system. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 2342–2348, New York, NY, USA. Association for Computing Machinery. find
  54. Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, and Norman Sadeh. 2019. Question answering for privacy policies: Combining computational and legal perspectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4947–4958, Hong Kong, China. Association for Computational Linguistics. ↗
  55. Behrooz Mansouri and Ricardo Campos. 2023. Falqu: Finding answers to legal questions. Preprint, arXiv:2304.05611. ↗
  56. Felicia Redelaar, Romy Van Drie, Suzan Verberne, and Maaike De Boer. 2024. Attributed question answering for preconditions in the Dutch law. In Proceedings of the Natural Legal Language Processing Workshop 2024, pages 154–165, Miami, FL, USA. Association for Computational Linguistics. find
  57. Meta. 2025. Llama-3.3-70b-instruct model card. https://huggingface.co/meta-llama/Llama-3. 3-70B-Instruct. Accessed: 2026-02-10. ↗
  58. Meta. 2026. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta.com/blog/ llama-4-multimodal-intelligence/. Accessed: 2026-03-27. ↗
  59. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. ↗
  60. Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, and 181 others. 2026. Kimi k2: Open agentic intelligence. Preprint, arXiv:2507.20534. ↗
  61. Stephen Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at trec-3. pages 0–.
  62. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  63. Francesco Sovrano, Monica Palmirani, Biagio Distefano, Salvatore Sapienza, and Fabio Vitali. 2021. A dataset for evaluating legal question answering on private international law. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, ICAIL ’21, page 230–234, New York, NY, USA. Association for Computing Machinery. find
  64. Stéphan Tulkens, Chris Emmery, and Walter Daelemans. 2016. Evaluating unsupervised Dutch word embeddings as a linguistic resource. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4130– 4136, Portorož, Slovenia. European Language Resources Association (ELRA). find
  65. Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11–21. ↗
  66. Stephan Tulkens and Thomas van Dongen. 2024. Model2vec: Fast state-of-the-art static embeddings. find
  67. Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and Han Xiao. 2025. Jina embeddings v3: Multilingual text encoder with low-rank adaptations. In Advances in Information Retrieval: 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, April 6–10, 2025, Proceedings, Part V, page 123–129, Berlin, Heidelberg. Springer-Verlag. find
  68. Santosh T.y.s.s., Rashid Haddad, and Matthias Grabmair. 2024. ECtHR-PCR: A dataset for precedent understanding and prior case retrieval in the European court of human rights. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5473–5483, Torino, Italia. ELRA and ICCL. ↗
  69. Weihang Su, Yiran Hu, Anzhe Xie, Qingyao Ai, Quezi Bing, Ning Zheng, Yun Liu, Weixing Shen, and Yiqun Liu. 2024. STARD: A Chinese statute retrieval dataset derived from real-life queries by non-professionals. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10658–10671, Miami, Florida, USA. Association for Computational Linguistics. find
  70. VoyageAI. 2024. Domain-specific embeddings and retrieval: Legal edition (voyage-law-2). VoyageAI blog. Accessed: 2025-11-13. find
  71. VoyageAI. 2025. Voyage 3 large. https://blog.voyageai.com/2025/01/07/voyage-3-large/. Accessed: 2025-11-13. ↗
  72. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. ↗
  73. Xin Sun, Jianan Xie, Zhongqi Chen, Qiang Liu, Shu Wu, Yuehe Chen, Bowen Song, Zilei Wang, Weiqiang Wang, and Liang Wang. 2025. Divide-then-align: Honest alignment based on the knowledge boundary of rag. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11461–11480. ↗
  74. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916.
  75. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025a. Gemma 3 technical report. Preprint, arXiv:2503.19786. ↗
  76. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024b. Multilingual e5 text embeddings: A technical report. Technical Report MSR-TR-2024-45, Microsoft. ↗
  77. GLM Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, and 152 others. 2025b. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. Preprint, arXiv:2508.06471. ↗
  78. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021. MiniLMv2: Multi-head self-attention relation distillation for compressing pre-trained transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2140–2151, Online. Association for Computational Linguistics. find
  79. Yizhen Wang, Xueying Shen, Zixian Huang, Lihui Niu, and Shiyan Ou. 2024c. clegal-qa: a chinese legal question answering with natural language generation methods. Complex & Intelligent Systems, 11. ↗
  80. Youchao Zhou, Heyan Huang, Yicheng Liu, Rui Dai, Xinglin Wang, Xingchen Zhang, Shumin Shi, and Yang Deng. 2026. Do retrieval augmented language models know when they don’t know? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 35158–35166. ↗
  81. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 641–649, New York, NY, USA. Association for Computing Machinery.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/banar-2026-blleqa-benchmarking-llms-grounded-legal/