Elamur Research⌕
☰
home / papers / 2026 / MizanQA: A Benchmark for Multi-Answer Moroccan Legal QA

MizanQA: A Benchmark for Multi-Answer Moroccan Legal QA

Paper · 2026 · Adil Bahaj and Mounir Ghogho

Summary

LLMs are still weak at legal question answering in Arabic, and Moroccan law is especially hard. Its texts are in Modern Standard Arabic but carry local idioms, and they draw on Maliki jurisprudence, customary rules and French legal tradition. The authors built MizanQA to test this. It has 1,776 expert-checked multiple-choice questions taken from public Moroccan law question banks and exams. Many questions have more than one correct option, and the number of options varies.

Construction mixed manual and automatic steps:

  • an expert removed sources that rely on outdated law;
  • pages were turned into image batches;
  • a multimodal LLM extracted the items;
  • curators verified and categorised them by hand.

The authors also designed metrics for multi-answer scoring: strict exact-set accuracy, a precision-penalised F1-like score, a partial-credit score with a penalty, and two calibration-error variants. Six models were tested zero-shot with an Arabic prompt and no retrieval. Gemini-2.0-Flash was the most accurate, and Llama-4-Maverick was the best calibrated. All models still showed large gaps, and scores fell as the number of options grew. The benchmark gives a parametric-knowledge baseline for future retrieval and reasoning tracks.

Key points

  • MizanQA has 1,776 questions in 9 law categories. Questions have 2–12 options and 1–10 correct answers. Criminal Law is the largest category (847 items). The data is released on Hugging Face as adlbh/MizanQA-v0.
  • Gemini-2.0-Flash did the item extraction from page images, and humans then checked the output against written guidelines.
  • The models tested were Allam-2 (7b), Gemini-1.5-Flash, Gemini-2.0-Flash, Llama-3.3 (70b), Llama-4-Maverick and Llama-4-Scout. Settings were zero-shot, temperature 1, with no tools or retrieval.
  • On strict accuracy, Gemini-2.0-Flash scored 42.11 and Allam-2 scored only 15.32. Llama-4-Maverick edged ahead on F1-like(1) (64.90 vs 64.84) and had the lowest ECE (17.64 per-option, 29.10 set-level).
  • Contracts and Constitution questions were answered better than Family Code and Criminal Law questions. More options led to lower scores, and set-level calibration error grew faster than the per-option variant.

Interesting ideas

  • I1. Partial-credit scoring is tunable: correct picks minus a weighted count of wrong picks, divided by the size of the gold set and clipped to [0,1], so guessing extra options costs points.
  • I2. An F1 variant weights false positives by a factor α inside precision, which lets the evaluator choose how much worse a wrong claim is than a missed one.
  • I3. Calibration is measured at two levels: per option, and per whole answer set using the product of option confidences. This shows how small per-option overconfidence compounds into large set-level miscalibration.
  • I4. Before building the benchmark, an expert filtered out sources tied to superseded legislation, so the gold answers do not go stale.
  • I5. Results are broken down by number of options to expose difficulty scaling.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-test readers that face "select all that apply" questions could be scored this way, so over-selection is punished separately from missed answers.
  • I2 (learn): Evaluating silver frames or role assignments with α>1 would favour precise, abstaining outputs over noisy recall.
  • I3 (world): Confidence over a whole extracted world (all entities, states and event order together) could be checked against exact correctness, not just per fact.
  • I4 (world): Compiled knowledge from Wikidata and similar sources could carry validity dates and be filtered the same way before it is used as ground truth.
  • I5 (answer): QA on fables could report accuracy against the number of candidate entities or answers, showing where reasoning degrades.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 4Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksQuestion answering · 441 worksLegal AI · 134 works

Cite

Cite as Adil Bahaj and Mounir Ghogho. 2026. MizanQA: A Benchmark for Multi-Answer Moroccan Legal QA. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.10

@inproceedings{bahaj2026mizanqa,
  title = {{MizanQA: A Benchmark for Multi-Answer Moroccan Legal QA}},
  author = {Adil Bahaj and Mounir Ghogho},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.10},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.10}
}

References

17 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, and 1 others. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. ↗
  2. Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, and Amani Al-Ghraibah. 2025. Can large language models predict the outcome of judicial decisions? arXiv e-prints, pages arXiv–2501.
  3. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1):64–93. ↗
  4. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E Shamout. 2025. Medarabiq: Benchmarking large language models on arabic medical tasks. arXiv preprint arXiv:2505.03427. ↗
  5. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, and 1 others. 2024. Lawbench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7933–7962.
  6. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  7. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36:44123–44279.
  8. Faris Hijazi, Somayah Alharbi, Abdulaziz AlHussein, Harethah Shairah, Reem Alzahrani, Hebah Al-shamlan, George Turkiyyah, and Omar Knio. 2024. Arablegaleval: A multitask benchmark for assessing arabic legal knowledge in large language models. In
  9. Chakib Lebaidi Ismail Mellouki. 2021. Issues of equivalence in the moroccan legal text. Journal of University Studies for Inclusive Research, pages 1456–1478. find
  10. Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, and 1 others. 2024. Legalagentbench: Evaluating llm agents in legal domain. arXiv preprint arXiv:2412.17259. ↗
  11. Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29. find
  12. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. ↗
  13. Nicholas Pipitone and Ghita Houir Alami. 2024. Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain. arXiv preprint arXiv:2408.10343. ↗
  14. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, and 1 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  15. Mikolaj Sitarz. 2022. Extending f1 metric, probabilistic approach. arXiv preprint arXiv:2210.11997. ↗
  16. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
  17. Meta Llama Team. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation — ai.meta.com. https://ai.meta.com/blog/llama-4-multimodal-intelligence/. [Accessed 05-05-2025]. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bahaj-2026-mizanqa-benchmark-multi-answer-moroccan/