📄 CIS-RG at QIAS 2025 Shared Task: Approaches for Enhancing Performance of LLM on Islamic Legal Reasoning and its Mathematical Calculations

Summary

This system paper for the QIAS 2025 shared task examines how well LLMs handle Islamic inheritance law (ʿilm al-mawārīth). The task is hard because it combines legal reasoning with multi-step fraction arithmetic. A model must identify eligible and blocked heirs, apply fixed-share rules (farāʾiḍ), handle residuary heirs, and then adjust the resulting fractions. The authors fine-tuned four models (Fanar, Llama, Gemini, Mistral). Their small training sets of 100 and 200 questions were split into two groups: cases with arithmetic and cases without. They also drafted a step-by-step Chain of Thought (CoT) template for few-shot prompting, but the CoT results were still incomplete at the time of writing. On the unlabeled 1,000-question multiple-choice test set, the commercial models stayed above 90%, while the open models reached only the mid-70s. A qualitative analysis of four test items suggests that how a question is worded matters more than its difficulty label. The paper also points out weaknesses of the multiple-choice format and argues for neuro-symbolic or agentic approaches, with human legal verification of the outputs.

Key points

• The test data consists of 1,000 unlabeled MCQs, each with six options (A–F), covering inheritance scenarios. The setup follows the earlier benchmark studies by Bouchekif (2024, 2025).
• The fine-tuned open models scored 76% (Fanar), 74% (Mistral) and 73% (LLaMA). Gemini 2.5 and o3 exceeded 90%.
• The fine-tuning subsets had 100 and 200 questions, divided into "no calculation" and "requires calculation" clusters.
• The CoT template walks through six steps:
  • list the heir types;
  • mark the fixed-share heirs;
  • mark the non-fixed (residuary) heirs;
  • mark the blocked heirs;
  • draw the conclusion;
  • count the heirs in each type.
• The authors deliberately placed two identically worded options (B and E) in one question. Three models failed to notice and answered E. Labelling questions as beginner or advanced showed no clear effect on results.

Interesting ideas

• I1. Break a domain calculation into a fixed checklist: first enumerate the participants, then sort them into fixed-share, remainder-taking and excluded groups, and only then do the arithmetic. This separates comprehension errors from calculation errors.
• I2. Plant a known flaw in a test item, such as two options with exactly the same text, and check whether the system notices. This works as a cheap negative control for shallow pattern matching.
• I3. Group evaluation and training items by the kind of difficulty (logical reasoning versus numeric computation) rather than by a nominal level, since the level label did not predict errors here.
• I4. The paper warns that multiple-choice items whose justification text is shared across several options make the gold answer ambiguous. Explanations should therefore be tied uniquely to the answer they support.
• I5. Residuary heirs are ranked by an explicit rule order: first by the direction of kinship, then by its degree, then by its strength. This is a deterministic precedence that a symbolic layer can apply exactly, which matches the authors' case for neuro-symbolic solving.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): the maths word-problem reader could follow the same staged order over world entities, so every step can be inspected and blamed separately.
• I2 (learn): add duplicated or contradictory distractors to evaluation sets, so a reader that picks an answer without truly discriminating between options gets flagged.
• I3 (learn): stratify scores into "relational/logic" and "arithmetic" buckets to see which pipeline stage is actually failing.
• I4 (answer): when building school-test or fable QA sets, check that each option's supporting evidence is distinct, so explanation-based grading stays meaningful.
• I5 (world): encode transfer and holding rules as ordered precedence criteria in compiled knowledge, so share distributions in a text world are computed by rules rather than guessed.