📄 Al-Jawaab at Qur'an QA 2023 Shared Task: Exploring Embeddings and GPT Models for Passage Retrieval and Reading Comprehension

Summary

The paper describes a system entered in the Qur'an QA 2023 Shared Task, which has two parts: ranking Qur'anic passages for a Modern Standard Arabic question (Task A) and extracting every answer span from a given passage (Task B). Both tasks include questions with no answer in the text. For Task A, the authors embedded questions and passages with OpenAI's "text-embedding-ada-002" and ranked passages by cosine similarity. A similarity threshold tuned on the dev set decides when to return "no answer." This approach scored below the BM25 baseline. For Task B, they gave GPT-4 a handcrafted Arabic prompt at temperature zero, cleaned the raw output, and kept only answers that could be located verbatim in the passage. This clearly beat a naive whole-passage baseline. The work shows that prompting a strong LLM, combined with strict grounding of its output, can be competitive for span extraction. It also shows that off-the-shelf dense embeddings may lose to lexical retrieval on specialised Arabic text.

Key points

• Task A data: the Qur'anic passage collection (QPC) has 1,266 thematic passages drawn from the 114 chapters. AyaTEC supplies 199 questions, and QRels lists 1,132 relevant passage-ids. Task B uses QRCD v1.2, which has 1,399 question–passage–answer triplets, about 15% of them unanswerable.
• Task A: the 1536-dimension ada-002 embeddings with a 0.85 threshold reached MAP 0.109 on dev and 0.064 on test. BM25 scored 0.170 and 0.090 on the same sets. On test, MRR was 0.161 against BM25's 0.226.
• Translating questions and passages into English with Google Translate and rerunning retrieval performed noticeably worse than working on the Arabic directly.
• Task B with GPT-4 reached pAP 0.470 on dev and 0.539 on test, compared with 0.255 and 0.327 for the NWPR baseline. Extra post-processing raised the test score to 0.546, roughly a 67% gain. GPT-3.5 did so poorly on dev that it was dropped, falling even below the naive baseline.
• The authors observed prompt-injection-like behaviour, where a question's wording made GPT-4 ignore the instructions. They also name API cost and lack of explainability as limitations.

Interesting ideas

• I1. Every answer the LLM generates is mapped back to start and end offsets in the passage, and any answer that cannot be found verbatim is thrown away as out-of-passage. This turns a free-form generator into a grounded span extractor.
• I2. "No answer" is produced in two ways: either the model abstains explicitly, or every proposed answer fails the grounding check. This gives a cheap, principled abstention rule.
• I3. The authors report honestly that the dense embedding retriever lost to BM25, and they compare against a trivial whole-passage baseline. Weak, simple baselines thus act as sanity floors for the method.
• I4. Output from a nondeterministic generator arrives in shifting formats, such as quoted text or bulleted lists, so it is normalised before matching. Only after that is it checked against the source.
• I5. Instructions placed in the prompt could be overridden by the content of the question itself. This argues for keeping task instructions and user-supplied text clearly separated.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): readers answering fable or test questions could keep only answers anchored to a concrete span or node in the text, which makes each answer traceable.
• I2 (learn): LLM teacher labels could be accepted only when they align with the source text, with "nothing survives alignment" recorded as a distinct abstain signal instead of being forced into silver data.
• I3 (learn): every evaluation could include such a negative control, for example returning the whole sentence or a lexical-overlap pick, so that gains are measured against a dumb floor.
• I4 (learn): teacher-tier outputs could go through a fixed normalisation step before agreement checks with witness parsers, so that formatting noise does not count as disagreement.
• I5 (learn): when prompting LLM teachers on raw sentences, the data could be delimited strictly and outputs that drift off-task could be flagged, so that injected text does not contaminate silver labels.