📄 MorAI at QIAS 2025: Collaborative LLM via Voting and Retrieval-Augmented Generation for Solving Complex Inheritance Problems

Summary

This paper targets the first subtask of QIAS 2025, a shared-task competition in which systems must answer Arabic multiple-choice questions on Islamic inheritance law. Each heir's share follows from fixed legal rules plus exact arithmetic, and a small change in family composition can change the whole outcome. The authors pair retrieval with a multi-model ensemble. About 3,065 solved inheritance fatwas from IslamWeb, stored as JSON, are indexed with TF-IDF. For each question, the three closest cases by cosine similarity are added to the prompt. Three models then each choose one letter from A–F:

• a LoRA-fine-tuned ALLaM-7B-Instruct-preview,
• Gemini-2.5-Flash,
• Deepseek-reasoner.

The majority answer wins, and Gemini decides when all three disagree. The ensemble reached 88.00% accuracy, above its strongest member (Gemini-2.5-Flash with RAG, 83.50%), and ranked 5th in the subtask. The results suggest that for rule-heavy legal reasoning, plain lexical retrieval and voting across diverse models can outperform single models and neural retrieval.

Key points

• The task data has roughly 20,000 training, 1,000 validation and 1,000 test MCQs, derived from IslamWeb fatwas and reviewed by four experts. Each question has six options and three difficulty levels (beginner, intermediate, advanced).
• Accuracy of the submitted runs:
  • Fanar-1-9B: 36.10%
  • base ALLaM-7B-Instruct-preview: 26.50%
  • fine-tuned ALLaM: 79.50%
  • Deepseek-chat: 50.90%
  • Qwen3-1.7B with RAG: 26.10%
  • Gemini-2.5-Flash: 78.10% (83.50% with RAG)
  • Deepseek-reasoner with RAG: 71.80%
  • collaborative voting: 88.00%
• Retrieval settings were compared over k ∈ {3, 5, 7} and TF-IDF vs. neural embeddings. TF-IDF with k=3 was chosen as the best trade-off of accuracy, prompt length and compute.
• ALLaM was fine-tuned with LoRA (r=8, alpha=16, dropout 0.05) for 300 steps at learning rate 5e-5. Output was capped at 20 new tokens.
• The prompt forces a single English letter with no explanation. When the three voters disagree, the system falls back to Gemini, which the authors admit makes the ensemble less neutral. They propose accuracy-weighted voting as future work.

Interesting ideas

• I1. Fetching a few already-solved cases by simple keyword overlap beat semantic embeddings on rule-governed problems, likely because exact terms (heir types, kinship words) carry the decisive signal.
• I2. Voting across heterogeneous models with a designated tie-breaker makes the ensemble robust, but the authors note the tie-breaker biases it and suggest weighting each voter by measured accuracy.
• I3. Forcing the model to emit exactly one letter from a stated set, with a tiny token budget, makes scoring trivial and removes parsing noise from evaluation.
• I4. Reporting every submission separately (base model, fine-tuned, zero-shot, with RAG, ensemble) shows how much each component adds.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): A maths word-problem solver could retrieve previously solved problems that share exact quantity and relation words, and use their worked frame structures as templates rather than fuzzy look-alikes.
• I2 (learn): Silver-data agreement among witness parsers could weight each parser's vote by its held-out LAS, and flag three-way splits as low-confidence instead of silently trusting one parser.
• I3 (learn): When an LLM teacher tier labels test items, a strict closed answer set and short output keep teacher labels machine-checkable and cheap to compare against Mova's own answers.
• I4 (learn): Evaluations could present a similar ladder (e.g., bare parser, plus repair rules, plus witness agreement) so each stage's contribution is visible next to negative controls.