📄 N&N at QIAS 2025: Chain-of-Thought Ensembles with Retrieval-Augmented framework for Classical Arabic Islamic

Summary

The authors address QIAS 2025 Subtask 2, a shared task in which systems answer Arabic multiple-choice questions drawn from classical Islamic literature. The questions span jurisprudence, the prophetic biography, Qur'anic sciences and Hadith studies, and are graded Beginner, Intermediate or Advanced. Their pipeline has three stages. First, several commercial LLMs answer each question using a standardized Arabic prompt with five chain-of-thought demonstrations taken from the validation set. Second, the three strongest models vote, and an answer is accepted only when at least two agree. Third, if no answer reaches that majority, a retrieval step searches the digitized reference books, reranks the retrieved passages and re-asks GPT-4o with the best ones as context. The combined system scores 89.8% accuracy on the hidden test set, higher than any single model. The work shows that agreement-gated ensembling, with retrieval used only when needed, is a cheap way to strengthen LLMs on specialized, culturally specific domains in languages other than English.

Key points

• Data: 1,400 four-option MCQs (700 validation, 700 test). Supplementary e-books such as Ar-Raḥīq al-Makhtūm and Al-Itqān fī Ulūm al-Qur'ān serve as the retrieval corpus.
• Candidate models were GPT-4o, Gemini 2.5-Flash, Qwen-Plus and DeepSeek-V3. Decoding used temperature 0.2, top-p 0.95 and a limit of 512 output tokens.
• Reported accuracies:
  • GPT-4o: about 0.85 / 0.85 / 0.86 across zero-, 3- and 5-shot.
  • Gemini 2.5: rose from 0.59 to 0.87 once examples were added.
  • DeepSeek-V3: reached 0.84 at 5-shot.
  • Qwen-Plus: reached 0.78 at 5-shot.
• The R²P retrieval module has three steps:
  • Dense search uses Arabic-LaBSE embeddings (768-d, mean-pooled, L2-normalized) with a FAISS IndexFlatIP index over chunks of roughly 200 tokens with overlap, returning the top 10 passages.
  • A hybrid BM25 plus cross-encoder reranker narrows these to 3 passages.
  • GPT-4o is re-prompted with those 3 passages.
• Gemini's weak zero-shot score came from ignoring the letter-only output instruction. Its long responses ran past the token limit and produced empty answers, and few-shot examples fixed this.

Interesting ideas

• I1. Retrieval runs only when the model ensemble disagrees, so the extra cost of evidence lookup is spent on the uncertain cases alone
• I2. A strict "at least two of three agree" threshold doubles as an uncertainty detector, not just a way to pick the final answer
• I3. Retrieval is split into two stages: a broad dense recall of top-10 passages, then a precise hybrid lexical-plus-neural rerank down to three
• I4. Format failures can masquerade as knowledge failures: Gemini's low zero-shot score was really about output compliance, and examples fixed it

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): School-test readers could fall back to searching compiled knowledge or the source text only when their internal reasoning paths give conflicting answers.
• I2 (learn): Silver data from witness parsers could use the same rule, accepting arcs only on 2-of-3 agreement and routing disagreements to an LLM teacher tier or to repair rules.
• I3 (world): Looking up WordNet/FrameNet/Wikidata facts for an entity could gather many candidates cheaply and then rerank them by lexical overlap and context fit before adding them to the text's world.
• I4 (learn): Evaluations using LLM teachers should separate malformed or truncated outputs from wrong answers, so a model is not judged weak because of formatting.