📄 TCE at Qur'an QA 2023 Shared Task: Low Resource Enhanced Transformer-based Ensemble Approach for Qur'anic QA

Summary

The paper describes the TCE team's systems for the two Qur'an QA 2023 shared tasks. Task A is ad hoc passage retrieval: questions are written in MSA, and the answers are passages of the Thematic Qur'an Passage Collection (QPC), which are in Classical Arabic. Task B is ranked extractive MRC over QRCDv1.2, in the style of SQuADv2.0. Training data is very small, so the authors combine several techniques:

• Pipelined fine-tuning of Arabic encoders (AraBERTv0.2-base, CAMeLBERT-CA, AraELECTRA), passing through external resources such as TyDi QA AR and Tafseer before the target task.
• Retrieval architectures for Task A: both dual-encoders and cross-encoders.
• Two span losses for Task B: first-answer and multi-answer.
• Self-ensembles that vote across runs to reduce run-to-run variance.
• Thresholds that decide when a question has no answer.

For Task B they also build leakage-aware "faithful" splits and apply overlap-removing post-processing. On the hidden test set, the best systems reach a MAP of 25.05% on Task A and a pAP of 57.11% on Task B, well above the baselines (9.03 and 32.68). The work matters because it shows how standard transformer pipelines can be stabilised for an extremely small, imbalanced QA benchmark.

Key points

• Task A design: dual-encoders were trained with STAR, using 16 queries per batch. Cross-encoders feed the question and passage to the model as one joint input. On the dev split, the best single model is a pipelined AraBERT cross-encoder (MAP 34.83%, MRR 47.09%), and its self-ensemble reaches a MAP of 36.70%.
• Hard negatives: mining hard negatives from a fine-tuned checkpoint improves the AraBERT dual-encoder by roughly 4.5 MAP points compared with random negatives.
• Multi-answer loss (MAL): on multi-answer samples, MAL clearly beats the first-answer loss (FAL). For example, AraELECTRA scores 51.55 vs 43.69 pAP on that subset. On the standard split overall, however, MAL does not always win.
• Post-processing: removing overlapping spans always improves on the raw span rankings, by about 3 pAP for AraBERT-MAL.
• Leakage: the authors extended the leakage analysis to QRCDv1.2. About 90% of its no-answer samples fall into the trivial leakage group D(1), which makes the dev scores on unanswerable cases unreliable.
• Released resources: code and trained models are on GitHub (mohammed-elkomy/quran-qa).

Interesting ideas

• I1. Leakage-aware splits: the authors re-split the data so that shared questions, passages or answers cannot sit on both sides of train and dev. This exposes how much of a score comes from memorisation rather than generalisation.
• I2. Multi-answer training loss: when a question has several correct spans, the loss spreads probability across all of them instead of rewarding only the first gold span. This keeps the model from becoming overconfident in a single answer.
• I3. Prior-calibrated abstention: the no-answer threshold is set so that the share of rejected items matches the expected rate of unanswerable questions (about 15% for Task A, about 5% for Task B). This is used instead of tuning the threshold on a tiny, unrepresentative dev set.
• I4. Self-ensemble voting: the same model is trained several times and the runs vote, which smooths out the variance that small data causes.
• I5. Hard negatives from a trained checkpoint: negatives for retrieval are mined using an already fine-tuned model, so the training signal sharpens around confusable passages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova's QA and parser evaluations could group items by shared text before splitting, as a negative control against inflated scores from near-duplicate fables or sentences.
• I2 (answer): Mova's readers could keep and score every grounded answer candidate in the world model, rather than forcing one winner when the text supports several.
• I3 (answer): Mova's "cannot answer from text" decision could be calibrated against a known base rate of unanswerable test questions, which makes abstention explicit and auditable.
• I4 (learn): this mirrors Mova's agreement among witness parsers; repeated gparse runs with different seeds could serve as extra witnesses when filtering silver trees.
• I5 (world): when Mova links mentions to WordNet or Wikidata entries, near-miss candidates proposed by an earlier linker could serve as hard negatives for training the disambiguation step.