📄 Gumball at QIAS 2025: Arabic LLM Automated Reasoning in Islamic Inheritance

Summary

The paper describes the winning entry of QIAS 2025 SubTask 1, which tests Islamic inheritance (fara'id) reasoning. Each item is a formal Modern Standard Arabic question with six answer options. Solving it requires legal rules, handling of exception cases and share arithmetic. The authors compare four ways of adapting a small, quantized Qwen3-4B model trained with Unsloth and LoRA:

• a RAG baseline over IslamWeb fatwas;
• plain SFT on the annotated MCQ set;
• instruction SFT followed by GRPO with custom rewards;
• a two-stage pipeline: continued pretraining on domain text, then instruction-style MCQ fine-tuning.

A deterministic Arabic normalization script runs before training and again on model outputs before scoring. The two-stage pipeline reached 97.2% test accuracy and took first place. The other approaches scored well below it. Most remaining errors come from advanced cases with many heirs, distant kin or share correction (tas'hih). The work shows that a modest model on one consumer GPU can master a rule-heavy legal domain when domain adaptation comes before task tuning.

Key points

• The data has 20,000 training questions and 1,000 each for dev and test, each split evenly between Beginner and Advanced. The answer labels are mildly skewed (C 21.7%, F 13.4%).
• Test accuracy by approach:
  • RAG: 35.33% in the summary table; the text reports a 44.0% best run with Arabic-all-nli-triplet-Matryoshka embeddings.
  • SFT + GRPO: 57%.
  • SFT on the annotated set: 87%.
  • Two-stage pipeline: 97.2%.
• GRPO applied directly to the base model gave only 15%. Starting GRPO from an MCQ-tuned checkpoint gave 57%.
• In a matched pair of runs, cleaning the evaluation data raised test accuracy from 94.3% to 95.1%.
• The error rate was 5.0% on Advanced questions versus 0.6% on Beginner ones. Training ran on a single RTX 3090 Ti (24GB).

Interesting ideas

• I1. Adapting the model to the domain's register through plain next-token training on domain prose, before teaching the task format, beat task-only tuning by about ten points.
• I2. The same deterministic normalizer is applied to training input and to model output before comparison with gold. This removes false mismatches that only reflect surface spelling.
• I3. The GRPO reward is split into separate checks: structural template, valid option format, exact numeric match and fuzzy partial credit. Each failure type then gets its own signal.
• I4. Reinforcement learning from a cold base model barely worked, while starting from a supervised checkpoint made it viable.
• I5. Errors are reported by difficulty tier and traced to specific phenomena (many heirs, distant kinship, share correction), not just as one overall score.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Before training a reader on school tests or fables, first expose the gparse encoder to unlabeled in-genre text so it learns that register.
• I2 (learn): Evaluation of answers and silver parses should canonicalize spelling, punctuation and number formats on both sides, so scores reflect meaning rather than orthography.
• I3 (answer): A maths word-problem solver could be scored separately on well-formed derivation, a legal answer type and the correct number, which shows where a wrong answer actually broke.
• I4 (learn): Self-play or repair-rule induction should start from a parser or reader already trained on silver data, not from scratch.
• I5 (world): Error reports for holdings and quantities in the text world could be grouped by structural complexity, such as the number of entities or transfers, to find where state tracking fails.