Elamur Research⌕
☰
home / papers / 2026 / QuALA-NL: Question & Answer with Legal Attribution in Dutch

QuALA-NL: Question & Answer with Legal Attribution in Dutch

Paper · 2026 · Romy A.N. van Drie, Roos M. Bakker, Daan L. Di Scala et al.

Summary

Legal applications of LLMs need answers that can be traced to their sources, and testing such systems requires datasets tied to a specific jurisdiction and language. The authors present QuALA-NL, a Dutch legal question-answering resource of 101 question–answer pairs drawn from three Dutch laws. Each pair is linked to the relevant chunks of statutory text and also to formal interpretations of those passages written in FLINT, an action-centred formalism built from Act, Fact and Duty frames. This double linking makes it possible to evaluate knowledge-based RAG (K-RAG), not only ordinary text retrieval. To show how the resource can be used, they compare four setups: LLM-only, RAG over law text, K-RAG over FLINT frames, and a hybrid of both. Retrieval works best on the formalization, but answers generated from plain law text score higher. A qualitative inspection suggests the generator does not yet exploit graph content well. The dataset, code and evaluation are public, so it can serve as a benchmark for structured-knowledge retrieval in law.

Key points

  • Dataset size and scope: 101 QA pairs covering the Participation Act, the Aliens Act and the National Old Age Pensions Act.
  • Choice of laws: the laws were picked for diversity, for having an existing FLINT formalization, and for how often people use or ask about them.
  • Relation to prior work: QuALA-NL extends Redelaar et al. (2024), which had 102 pairs over 25 laws and only precondition questions. QuALA-NL covers fewer laws but goes beyond preconditions.
  • FLINT frame structure: Act frames list an actor, action, object, recipient, precondition and postcondition. Duty frames add a duty holder and a claimant. Fact frames describe states of the normative system.
  • Evaluation and results:
    • Retrieval is scored with hit rate, precision and recall.
    • Generation is scored with ROUGE, METEOR, MAUVE and G-EVAL (fluency, consistency, coherence, relevance).
    • K-RAG retrieves best, while text-based RAG generates best.
    • The KG retriever is language-model based and works at the subgraph level.

Interesting ideas

  • I1. Each answer is linked both to raw source text and to a structured interpretation of that text, so one gold set can test text retrieval and structure retrieval side by side.
  • I2. The four-way comparison (no retrieval, text only, structure only, both) cleanly separates what each knowledge source contributes.
  • I3. The finding that structure helps retrieval but not generation shows a gap between finding the right facts and verbalizing them.
  • I4. FLINT treats law as actions with an actor, object, recipient, preconditions and postconditions that create or end facts. This is essentially an event model with state changes.
  • I5. The prompt forces a fixed answer marker plus a list of the document IDs actually used, which makes the model's attribution machine-checkable against gold sources.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Fable/tale QA items could store the supporting sentence and the supporting frame/world facts, letting us tell whether a wrong answer came from bad parsing or bad reasoning.
  • I2 (learn): The same ablation grid could serve as a control set, running the answerer with the tree only, the world only, both, and neither, to show which layer actually earns the score.
  • I3 (back): Generation from the meaning graph should be evaluated separately from fact selection, so text realization is not blamed on, or hidden by, a correct graph lookup.
  • I4 (world): Act-frame style pre/postconditions could be used to update entity states and holdings in the text world, making the event order and its consequences explicit.
  • I5 (learn): When querying LLM teacher tiers, requiring cited sentence or node IDs would let silver labels be checked automatically against the source span.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 1Research subjects and languages the work is about.Legal AI · 134 works

Cite

Cite as Romy A.N. van Drie, Roos M. Bakker, Daan L. Di Scala and Maaike de Boer. 2026. QuALA-NL: Question & Answer with Legal Attribution in Dutch. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5i9bqybga69e

@inproceedings{drie2026quala,
  title = {{QuALA-NL: Question \& Answer with Legal Attribution in Dutch}},
  author = {Romy A.N. {van Drie}, Roos M. Bakker, Daan L. {Di Scala} and Maaike {de Boer}},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5i9bqybga69e},
  url = {https://doi.org/10.63317/5i9bqybga69e}
}

References

50 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tara Athan, Guido Governatori, Monica Palmirani, Adrian Paschke, and Adam Wyner. 2015. Legalruleml: Design principles and foundations. In Reasoning Web International Summer School, pages 151–188. Springer. find
  2. Roos Bakker, Romy van Drie, Maaike de Boer, Robert van Doesburg, and Tom van Engers. 2022a. Semantic role labelling for dutch law texts. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 448–457. find
  3. Roos M Bakker, Maaike HT de Boer, Romy AN van Drie, and Daan Vos. 2022b. Extracting structured knowledge from dutch legal texts: A rule-based approach. In EKAW (Companion). find
  4. Roos M. Bakker, Akke J. Schoevers, Romy A. N. van Drie, Marijn P. Schraagen, and Maaike H. T. de Boer. 2025. Semantic role extraction in law texts: a comparative analysis of language models for legal information extraction. Artificial Intelligence and Law. ↗
  5. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  6. Jeroen Breteler, Thom van Gessel, Giulia Biagioni, and Robert van Doesburg. 2023. The flint ontology: an actor-based model of legal relations. In Knowledge Graphs: Semantics, Machine Learning, and Languages, pages 227–234. IOS Press. ↗
  7. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051. ↗
  8. Ruixi Chen. 2025. Retrieval-augmented generation with knowledge graphs: A survey. In Computer Science Undergradaute Conference 2025@ XJTU. find
  9. Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations. find
  10. Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre FT Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, et al. 2024. Saullm-7b: A pioneering large language model for law. arXiv preprint arXiv:2403.03883. ↗
  11. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130. ↗
  12. VAH Firdaus, PY Saputra, and D Suprianto. 2020. Intelligence chatbot for indonesian law on electronic information and transaction. In IOP Conference Series: Materials Science and Engineering, volume 830, page 022089. IOP Publishing. find
  13. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SI-GIR Conference on Research and Development in Information Retrieval, pages 2288–2292. ↗
  14. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627. ↗
  15. Cristine Griffo, João Paulo A Almeida, and Giancarlo Guizzardi. 2018. Conceptual modeling of legal relations. In International Conference on Conceptual Modeling, pages 169–183. Springer. find
  16. Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. Advances in Neural Information Processing Systems, 37:132876–132907.
  17. Rinke Hoekstra, Joost Breuker, Marcello Di Bello, Alexander Boer, et al. 2007. The lkif core ontology of basic legal concepts. LOAIT, 321:43–63. find
  18. Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao. 2025. Grag: Graph retrieval-augmented generation. find
  19. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906. ↗
  20. Soha Khazaeli, Janardhana Punuru, Chad Morris, Sanjay Sharma, Bert Staub, Michael Cole, Sunny Chiu-Webster, and Dhruv Sakalley. 2021. A free format legal question answering system. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 107–113. find
  21. Valentina Leone. 2021. Legal knowledge extraction in the data protection domain based on ontology design patterns. ↗
  22. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
  23. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  24. Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452. ↗
  25. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634. ↗
  26. Jorge Martinez-Gil. 2023. A survey on legal question–answering systems. Computer Science Review, 48:100552. ↗
  27. Nicholas Matsumoto, Jay Moran, Hyunjun Choi, Miguel E Hernandez, Mythreye Venkatesan, Paul Wang, and Jason H Moore. 2024. Kragen: a knowledge graph-enhanced rag framework for biomedical problem solving using large language models. Bioinformatics, 40(6):btae353. ↗
  28. Costas Mavromatis and George Karypis. 2024. Gnn-rag: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139. ↗
  29. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316. ↗
  30. Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2024. Graph retrieval-augmented generation: A survey. arXiv preprint arXiv:2408.08921. ↗
  31. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816–4828. ↗
  32. Marc Queudot, Éric Charton, and Marie-Jean Meurs. 2020. Improving access to justice with legal chatbots. Stats, 3(3):356–375. ↗
  33. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331. ↗
  34. Felicia Redelaar, Romy Van Drie, Suzan Verberne, and Maaike De Boer. 2024. Attributed question answering for preconditions in the dutch law. In Proceedings of the natural legal language processing workshop 2024, pages 154–165. find
  35. Diego Sanmartin. 2024. Kg-rag: Bridging the gap between knowledge and creativity. arXiv preprint arXiv:2405.12035. ↗
  36. Vorada Socatiyanurak, Nittayapa Klangpornkun, Adirek Munthuli, Phongphan Phienphanich, Lalin Kovudhikulrungsri, Nantawat Saksakulkunakorn, Phonkanok Chairaungsri, and Charturong Tantibundhit. 2021. Law-u: Legal guidance through artificial intelligence chatbot for sexual violence victims and survivors. IEEE Access, 9:131440– 131461. ↗
  37. Robert van Doesburg. 2017. A formal method for interpretation of sources of norms. Technical report, Leibniz Center for Law, University of Amsterdam. find
  38. Thom van Gessel, Giulia Biagioni, Jeroen Breteler, Ioannis Tolios, and Erik Boertjes. 2023. A toolset for normative interpretations in flint. In SEMAN-TiCS (Posters & Demos). Thom van Gessel and Frank Goossens. Formalizing flint: A logic of preconditions and consequences of actions. 17 DEON, page 407. find
  39. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. ↗
  40. Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE transactions on knowledge and data engineering, 29(12):2724–2743. ↗
  41. G Wiggers. 2023. The relevance of impact: bibliometric-enhanced legal information retrieval. Ph.D. thesis, Leiden University. find
  42. Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. arXiv preprint arXiv:2202.13296. ↗
  43. Xiangrong Zhu, Yuexiang Xie, Yi Liu, Yaliang Li, and Wei Hu. 2025. Knowledge graph-guided retrieval augmented generation. arXiv preprint arXiv:2502.06864. 9. ↗
  44. Language Resource References Chalkidis, Ilias and Jana, Abhik and Hartung, Dirk and Bommarito, Michael and Androutsopoulos, Ion and Katz, Daniel Martin and Aletras, Nikolaos. 2021. LexGLUE: A benchmark dataset for legal language understanding in English. ↗
  45. Chen, Andong and Yao, Feng and Zhao, Xinyan and Zhang, Yating and Sun, Changlong and Liu, Yun and Shen, Weixing. 2023. EQUALS: A real-world dataset for legal question answering via reading chinese laws. find
  46. Hoppe, Christoph and Pelkmann, David and Migenda, Nico and Hötte, Daniel and Schenck, Wolfram. 2021. Towards intelligent legal advisors for document retrieval and question-answering in German legal documents. IEEE. find
  47. Louis, Antoine and van Dijck, Gijs and Spanakis, Gerasimos. 2023. Interpretable long-form legal question answering with retrieval-augmented large language models.
  48. Mansouri, Behrooz and Campos, Ricardo. 2023. FALQU: Finding Answers to Legal Questions. ↗
  49. van Drie, Romy AN and de Boer, Maaike HT and Bakker, Roos M and Tolios, Ioannis and Vos, Daan. 2023. The Dutch Law as a Semantic Role Labeling Dataset. find
  50. Zhong, Haoxi and Xiao, Chaojun and Tu, Cunchao and Zhang, Tianyang and Liu, Zhiyuan and Sun, Maosong. 2020. JEC-QA: a legal-domain question answering dataset. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/drie-2026-quala-nl-question-answer-legal/