📄 Candidate-Aware Retrieval and Reranking for Multiple-Choice Question Answering: Arabic as a Case Study

Summary

The paper targets multiple-choice QA in Arabic, where LLMs score well below their English level and plain RAG often adds noise or passages that are on-topic but do not settle the question. Instead of retrieving on the question alone, the authors build one query per answer option by joining the question with that option. They retrieve top-k passages for each query from a Milvus/Zilliz Cloud index. A lightweight reranker, trained with a combined regression and triplet loss, then picks the most supportive passage for each option. The chosen passages are concatenated into an extended ArabicMMLU prompt.

The work adds two resources. HumCorpus is a retrieval base built from Arabic textbooks and general-knowledge texts. RankMCQ, derived from PalmX 2025 Islamic Culture data, holds passages labelled positive, partially positive and negative. Evaluation covers ArabicMMLU humanities questions at high-school and university level, using small models (Llama-3.2, Qwen3, Mistral). The method beats both a standard RAG setup and a baseline reranker on average, and Qwen3-4B comes close to much larger systems. This shows that evidence selection aware of each answer option can stand in for model scale.

Key points

• Corpus construction. HumCorpus has these chunk counts:

  • Islamic Studies: 257,084
  • History: 49,833
  • Law: 16,806
  • Philosophy: 8,463

  Chunks are at most 1,000 tokens with a 200-token overlap. They are embedded with an Arabic Sentence-BERT model (768 dimensions) and searched by cosine similarity.

• Reranker supervision. RankMCQ gives each question three kinds of passage:

  • one that resolves it,
  • one that is relevant but insufficient,
  • one that is false or misleading.

  The data came from a two-stage pipeline that uses LLM generation.

• Average zero-shot accuracy (Qwen3-4B): 57.5 for this method, 56.4 for the baseline reranker, 55.2 for standard RAG and 53.7 with no retrieval.

• Mistral-7B: accuracy rises from 31.3 to 47.3. In Philosophy it jumps from 20.5 to 61.5.

• Comparison with larger models. Qwen3-4B with this method reaches 74.3 in Philosophy against 74.4 for GPT-4. The authors note that the reference baselines also include easier education levels.

• Ablation. Adding the triplet/contrastive term gives lower and faster-falling validation loss than regression alone.

Interesting ideas

• I1. Build one retrieval query per answer option (question plus that option) and keep the best evidence for each option, so every option is tested against its own support or refutation.
• I2. Use three graded evidence labels (resolving, relevant-but-insufficient, misleading) rather than a binary relevant/irrelevant split, which targets the common "on-topic but not decisive" failure.
• I3. Combine a regression score with a triplet margin loss, so the model gets calibrated relevance values and also a clear separation between close classes.
• I4. Report the cases where added context hurts a model, alongside average gains, as a measure of how robust the method is.
• I5. Offload vector storage and search to a managed service so the reranker can run on the same modest local hardware.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): School-test readers could look up evidence separately for each option in the world of the text or the knowledge graph, then compare the per-option justifications instead of scoring the question alone.
• I2 (learn): Evaluation sets and negative controls could include near-miss distractor passages, to check that a reader really needs the decisive fact and is not just matching the topic.
• I3 (learn): This objective could suit any learned scorer that must rank near-identical candidates, such as choosing among competing parses or frame readings proposed by several witnesses.
• I4 (answer): The QA harness could log every question where adding retrieved knowledge flips a correct answer to a wrong one, showing where compiled knowledge interferes.
• I5 (world): Large lookup structures for the knowledge layer (Wikidata or WordNet indices) could sit behind a separate service, leaving local GPU time for parsing with gparse.