From Query to Counsel: Structured Reasoning with a Multi-Agent Framework and Dataset for Legal Consultation

Paper · 2026 · Mingfei Lu, Yi Zhang, Mengjia Wu et al.

Summary

The paper targets legal consultation question answering (Legal CQA). In this task, ordinary users describe personal legal problems in vague, informal language, and the system must return grounded, actionable advice under Chinese law. The authors identify three obstacles: scarce realistic training data, tasks made of several interlinked subtasks, and heavy reliance on context. To address the data gap, they build JurisCQAD, a collection of more than 43,000 real consultation queries from public forums, each paired with expert-checked good and bad answers. They also propose JurisMA, a multi-agent pipeline with three phases:

An Element Agent first converts each query into a JSON legal element graph.
A Draft Agent writes an initial answer.
A Manager Agent repeatedly sends the draft to formatting and statute-retrieval agents before a final content check.

Models are trained on JurisCQAD using DPO and tested on a corrected version of LawBench. JurisMA beats both general and law-specific LLMs on lexical and semantic metrics. This suggests that explicit decomposition and coordinated agents make legal advice more reliable and easier to interpret.

Key points

JurisCQAD: over 43K anonymized Chinese consultations stored as (question, positive answer, negative answer) triplets. It is released for non-commercial research under an academic license.
Element graph: nodes represent entities, events, the claims the user makes, the essential facts, and the legal issues deduced from the query. Edges encode relations such as kinship or contractual duties.
JurisMA agents: Element, Draft, Manager, FormatCheck, LawSearch and ContentCheck. The Manager decides each round whether more refinement is needed.
Full-system scores on corrected LawBench:

Model ROUGE-L BLEU-N BERTScore

Qwen2.5:7b 28.01 11.32 73.82

Qwen2.5:14b 31.14 14.25 75.05
Ablations: removing the revision stage hurts most (14B: ROUGE-L −11.18, BERTScore −5.49). Dropping the graph or the Manager also lowers scores. Appendix experiments report gains on cross-lingual, cross-task and cross-jurisdiction benchmarks.

Model	ROUGE-L	BLEU-N	BERTScore
Qwen2.5:7b	28.01	11.32	73.82
Qwen2.5:14b	31.14	14.25	75.05

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Institutions

🎓University of Technology SydneySydney ·

Australia18 works · 12,833 citations 🎓University of BirminghamBirmingham ·

United Kingdom25 works · 62,564 citations

Fields, topics and concepts

📚 Shelves 7Computer Science · 16,667 works Artificial Intelligence · 16,194 works Arts and Humanities · 16,039 works Language and Linguistics · 15,891 works Social Sciences · 1,772 works Political Science and International Relations · 182 works Law · 77 works

🧭 Topics 2Reasoning · 1,426 works Legal AI · 134 works

Cite

Cite as Mingfei Lu, Yi Zhang, Mengjia Wu and Yue Feng. 2026. From Query to Counsel: Structured Reasoning with a Multi-Agent Framework and Dataset for Legal Consultation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.776

@inproceedings{lu2026query,
  title = {{From Query to Counsel: Structured Reasoning with a Multi-Agent Framework and Dataset for Legal Consultation}},
  author = {Mingfei Lu, Yi Zhang, Mengjia Wu and Yue Feng},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.776},
  url = {https://doi.org/10.18653/v1/2026.acl-long.776}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Jiaxi Cui, Munan Ning, Zongjian Li, Bohua Chen, Yang Yan, Hao Li, Bin Ling, Yonghong Tian, and Li Yuan. 2024. Chatlaw: A multi-agent collaborative legal assistant with knowledge graph enhanced mixture-of-experts large language model. Preprint, arXiv:2306.16092. ↗
Wentao Deng, Jiahuan Pei, Keyi Kong, Zhe Chen, Furu Wei, Yujun Li, Zhaochun Ren, Zhumin Chen, and Pengjie Ren. 2023. Syllogistic reasoning for legal judgment analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13997–14009, Singapore. Association for Computational Linguistics. find
Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. 2024. LawBench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7933–7962, Miami, Florida, USA. Association for Computational Linguistics. ↗
Herbert Lionel Adolphus Hart. 2012. The concept of law. OUP Oxford. ↗
Wanwei He, Jiabao Wen, Lei Zhang, Hao Cheng, Bowen Qin, Yunshui Li, Feng Jiang, Junying Chen, Benyou Wang, and Min Yang. 2023. Hanfei-1.0. https://github.com/siat-nlp/HanFei. ↗
Quzhe Huang, Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023. Lawyer llama technical report. Preprint, arXiv:2305.15062. ↗
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. ↗
Lasal Jayawardena, Nirmalie Wiratunga, Ramitha Abeyratne, Kyle Martin, Ikechukwu Nkisi-Orji, and Ruvan Weerasinghe. 2024. Scale-qa: Sri lankan case law embeddings for legal qa. find
Hans Kelsen. 1967. Pure theory of law. Univ of California Press. ↗
Haitao Li, Qingyao Ai, Qian Dong, and Yiqun Liu. 2024. Lexilaw: A scalable legal language model for comprehensive legal understanding. find
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2024. Interpretable long-form legal question answering with retrieval-augmented large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22266–22275.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International conference on machine learning, pages 11328–11339. PMLR. ↗
Yixiao Ma, Yueyue Wu, Weihang Su, Qingyao Ai, and Yiqun Liu. 2023. Caseencoder: A knowledge-enhanced pre-trained model for legal case encoding. arXiv preprint arXiv:2305.05393. ↗
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. ↗
Shiwen Ni, Hao Cheng, and Min Yang. 2025. Pre-training, fine-tuning and re-ranking: A three-stage framework for legal question answering. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
Yu Zhang, Shutong Qiao, Jiaqi Zhang, Tzu-Heng Lin, Chen Gao, and Yong Li. 2025. A survey of large language model empowered agents for recommendation and search: Towards next-generation information retrieval. arXiv preprint arXiv:2503.05659. ↗
SK Nigam, SK Mishra, AK Mishra, N Shallum, and A Bhattacharya. 2023. Legal question-answering in the indian context: Efficacy, challenges, and potential of modern ai models. arxiv, 1–15. find
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗
Stuart J Russell and Peter Norvig. 2016. Artificial intelligence: a modern approach. pearson. find
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696. ↗
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020a. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5218–5230, Online. Association for Computational Linguistics.
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020b. Jec-qa: a legal-domain question answering dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9701–9708. find
Zhi Zhou, Jiang-Xin Shi, Peng-Xiao Song, Xiao-Wen Yang, Yi-Xuan Jin, Lan-Zhe Guo, and Yu-Feng Li. 2024. Lawgpt: A chinese legal knowledge-enhanced large language model. Preprint, arXiv:2406.04614. ↗
Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Ken Satoh, Min Zhang, and Shaoping Ma. 2020. Bert-pli: Modeling paragraph-level interactions for legal case retrieval. In IJCAI, volume 2020, pages 3501–3507. find
Yiquan Wu, Yuhang Liu, Yifei Liu, Ang Li, Siying Zhou, and Kun Kuang. 2024. wisdominterrogatory. Available at GitHub. find
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
Rujing Yao, Yang Wu, Chenghao Wang, Jingwei Xiong, Fang Wang, and Xiaozhong Liu. 2025. Elevating legal llm responses: Harnessing trainable logical structures and semantic knowledge with legal reasoning. arXiv preprint arXiv:2502.07912. ↗
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).

Updated 1 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2026-query-counsel-structured-reasoning-multi/