📄 ThinkDrill at IslamicEval 2025 Shared Task: LLM Hybrid Approach for Qur'an and Hadith Question Answering

Summary

The paper addresses Subtask 2 of IslamicEval 2025. Given a question in Modern Standard Arabic, a system must rank answer-bearing passages drawn from the whole Qur'an (1,266 QPC passages) and Sahih Bukhari (2,254 hadiths), or report that no answer exists. Earlier Qur'an QA editions supplied a gold passage and covered only the Qur'an; this task removes the passage and adds hadith.

The team built three retrieval pipelines and merged them:

• a multilingual-e5-base retriever fine-tuned with triplet loss and refined by an MS MARCO cross-encoder reranker;
• GPT-4 keyword extraction followed by RapidFuzz fuzzy matching;
• Claude Sonnet 4 prompted to return exact verses or hadiths, or -1.

To expand the training data, they added hadith answers by hand and aligned the external HAQA dataset to Bukhari narrations. Pipeline scores were normalized and combined by weighted averaging. The submission reached MAP@10 of 0.2296 on test (0.32 on dev). The hybrid outperformed each individual pipeline, which suggests that dense, lexical and LLM signals complement each other when retrieving religious texts.

Key points

• Test scores: MAP@10 0.2296, MAP_Q@5 0.2623, MAP_H@5 0.215. On dev, MAP@10 was 0.32 and MAP_Q@5 was 0.35.
• Dev ablation (MAP@10): embedding only 0.15, keyword only 0.08, LLM only 0.12, hybrid 0.173. The paper's reporting is inconsistent: the conclusion also calls 0.173 the final benchmark score.
• Models: intfloat/multilingual-e5-base was fine-tuned for 2 epochs with batch size 16, learning rate 2e-5, 100 warm-up steps and cosine-distance triplet loss. The reranker was cross-encoder/ms-marco-MiniLM-L-6-v2.
• Preprocessing:
  • Arabic cleanup removed diacritics, tatweel and honorifics and normalized letter variants; stopwords were kept.
  • Long passages were split into 150-character chunks with a 30-character overlap.
  • Fuzzy matching used a partial-ratio threshold of 70%.
• Coverage and no-answer detection: the test set had 71 questions, 17 of them unanswerable, and coverage was 76.1%. A top-score threshold of 0.35 flagged no-answer questions with precision 0.65 and recall 0.47.

Interesting ideas

• I1. Unanswerable questions enter triplet training with a "no answer" placeholder as the positive and every real passage as a negative, so abstention is learned directly rather than bolted on afterwards.
• I2. An external QA set (HAQA) was made usable by normalizing both sides, fuzzy-aligning its answers to the target corpus and keeping only matches above a similarity threshold.
• I3. The LLM is restricted to quoting exact source text or answering -1, which makes its output checkable by string lookup and limits hallucinated content.
• I4. Each component was scored alone on dev and then compared with the weighted combination, which shows exactly how much every signal contributes.
• I5. The error analysis separates lexical overlap with a different intent (prayer times vs. prayer importance) from genuine relevance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): fable and school-test readers could treat "not stated in the text" as an explicit candidate that competes with evidence-backed answers under a calibrated threshold.
• I2 (learn): silver data could be built the same way by aligning outside resources to our corpora, keeping only high-confidence alignments and checking a sample with negative controls.
• I3 (learn): LLM teacher tiers could be required to return verbatim spans from the input, so each label can be verified mechanically against the text before it is accepted.
• I4 (learn): when merging witness parsers or teachers, report each one's standalone score beside the ensemble so a weak witness is visible rather than hidden in the average.
• I5 (frames): question frames should mark the asked-about role or aspect, so that answers sharing words but filling a different frame slot are rejected.