Elamur Research⌕
☰
home / papers / 2025 / Few-Shot Prompting for Extractive Quranic QA with Instruction-Tuned LLMs

Few-Shot Prompting for Extractive Quranic QA with Instruction-Tuned LLMs

Summary

This work addresses extractive question answering over the Qur'an, using Task B of the Qur'an QA 2023 Shared Task. Given a Quranic passage and a question in Modern Standard Arabic, a system must return one or more exact sub-spans of the passage, or nothing when no answer exists. Classical Arabic makes this hard: rich morphology, free word order, metaphor, missing diacritics, and themes that span several chapters. The authors compare two routes. The first fine-tunes Arabic encoders (AraBERT, AraELECTRA, CAMeLBERT), starting from TyDi QA checkpoints, on a merged corpus of QUQA, ARCD and QRCDv1.2. The second gives Gemini and DeepSeek three-shot prompts written in Arabic. Both routes feed a post-processing stage that aligns subwords, suppresses overlapping spans, and filters vague or redundant answers. Prompted Gemini scores best, at pAP@10 0.637, which is above the shared-task winner. This suggests that careful prompting with span-faithful cleanup can beat fine-tuning in a low-resource domain with dense meaning.

Key points

  • The benchmark is QRCDv1.2 from the Qur'an QA 2023 Shared Task (Task B). Roughly 81% of its question–passage pairs have a single answer, about 14% have several, and about 5% have none.
  • Fine-tuned models were built with SimpleTransformers from TyDi QA checkpoints and trained on QUQA, ARCD and QRCDv1.2 (lr 2e-5, 10 epochs, batch size 16). The best was "bert-base-arabertv02-tydiqa" at pAP@10 0.503, which would place seventh in the standings.
  • Gemini with three-shot Arabic prompts reached pAP@10 0.637. Prior Task B entries scored lower: Tce 0.571 (ranked first), Al-Jawaab 0.539 (GPT-4 span extraction after dense retrieval), LKAU23 0.49, and GYM 0.461.
  • The few-shot demonstrations were taken from QRCDv1.2 and deliberately included single-answer, multi-answer and zero-answer cases.
  • Post-processing has four parts: subword alignment, non-maximum suppression of overlapping spans, semantic filtering, and re-ranking. The authors credit it with improving precision and reducing hallucinated answers.

Interesting ideas

  • I1. An LLM's free-text output is accepted only once it can be mapped back to an exact stretch of the source passage, which turns a generative model into a strictly extractive one and cuts hallucination.
  • I2. Candidate answers that overlap are pruned with non-maximum suppression, so near-duplicate spans with slightly different boundaries do not crowd the top of the ranked list.
  • I3. The few-shot demonstrations are chosen to cover the full range of answer types, including questions with no answer, so the model learns that answering "nothing" is a valid option.
  • I4. Several heterogeneous QA datasets are first rewritten into one common schema and then pooled, so that general-domain Arabic data (ARCD) can support a niche domain.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): every answer a reader produces could be required to resolve to concrete token offsets in the text, rejecting outputs with no source span and keeping each answer explainable.
  • I2 (answer): when several frames or nodes support overlapping answer spans in a tale QA item, NMS-style pruning could keep the single best-bounded span and drop the redundant variants.
  • I3 (learn): prompts for the LLM teacher tiers could always include unanswerable examples, giving built-in negative controls and discouraging the teacher from inventing labels.
  • I4 (tree): the same approach fits UD dialect conversion, where treebanks with different conventions are normalized into one target format before being pooled for gparse or the perceptron parser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Language models & pretraining Question answering Semitic languages

 Institutions

🎓October University of Modern Sciences and ArtsCairo · Egypt7 works · 25 citations

Cite

Cite as Mohamed Basem, Islam Oshallah, Ali Hamdi and Ammar Mohammed. 2025. Few-Shot Prompting for Extractive Quranic QA with Instruction-Tuned LLMs. arXiv:2508.06103.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2508.06103 · arXiv 2508.06103

@misc{basem2025few,
  title = {{Few-Shot Prompting for Extractive Quranic QA with Instruction-Tuned LLMs}},
  author = {Mohamed Basem, Islam Oshallah, Ali Hamdi and Ammar Mohammed},
  year = {2025},
  doi = {10.48550/arxiv.2508.06103},
  eprint = {2508.06103},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.06103}
}

References

20 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Alrayzah, F. Alsolami, and M. Saleh, “Challenges and opportunities for Arabic question-answering systems: current techniques and future directions,” PeerJ Computer Science, vol. 9, p. e1633, 2023. find
  2. R. Alnefaie et al., “Lkau23 team: Transformer-based ensemble for Qur’anic QA,” in Proc. 1st Workshop on Question Answering on the Quran, pp. 141–147, 2023. find
  3. TheDevastator, “ARCD Dataset: Unlocking Arabic Language Comprehension with the Arabic Reading Comprehension Dataset,” 2021. [Online]. Available: https://www.kaggle.com/datasets/thedevastator/ unlocking-arabic-language-comprehension-with-the/data ↗
  4. M. Basem, I. Oshallah, B. Hikal, A. Hamdi, and A. Mohamed, “Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models,” arXiv preprint arXiv:2412.11431, 2024. ↗ our card
  5. A. Elkomy, M. Elsayed, and M. Hefny, “TCE team: Transformer-based ensemble for extractive Qur’anic QA,” in Proc. 1st Workshop on Question Answering on the Quran, pp. 110–117, 2023. find
  6. M. Essam et al., “Decoding Queries: An In-Depth Survey of Quality Techniques for Question Analysis in Arabic Question Answering Systems,” IEEE Access, 2024. find
  7. M. U. Hadi et al., “Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects,” Authorea Preprints, vol. 1, pp. 1–26, 2023. find
  8. S. Kazi, S. Khoja, and A. Daud, “A survey of deep learning techniques for machine reading comprehension,” Artificial Intelligence Review, vol. 56, no. Suppl 2, pp. 2509–2569, 2023. find
  9. A. Liu, B. Feng, et al., “DeepSeek-V3 Technical Report,” arXiv preprint arXiv:2412.19437, 2025. [Online]. Available: https://arxiv.org/abs/2412. 19437 ↗
  10. M. Mahmoudi, M. Elaraby, and A. Mourad, “Gym team: Multi-task transfer learning for semantic span detection in Qur’anic question answering,” in Proc. 1st Workshop on Question Answering on the Quran, pp. 134–140, 2023. find
  11. M. Malhas, H. Mubarak, and K. Darwish, “Qur’an QA 2023: Overview of the First Workshop on Question Answering on the Quran,” in Proc. 1st Workshop on Question Answering on the Quran, pp. 1–13, 2023. [Online]. Available: https://aclanthology.org/2023.quran-1.1 ↗
  12. I. Oshallah, M. Basem, A. Hamdi, and A. Mohammed, “Cross-Language Approach for Quranic QA,” arXiv preprint arXiv:2501.17449, 2025. ↗
  13. B. I. OsmanHamoud, “A Question Answering System Design about the Holy Quran,” Ph.D. dissertation, Sudan Univ. of Science and Technology, 2017. find
  14. H. S. Al-Khalifa and N. Alnazzawi, “QUQA: Question Answering Dataset for the Holy Qur’an,” 2023. [Online]. Available: https://github.com/scsaln/HAQA-and-QUQA/blob/main/QUQA.xlsx ↗
  15. bigIR Research Group, “Qur’an QA 2023 Shared Task - Task B Dataset,” 2023. [Online]. Available: https://gitlab.com/bigirqu/quran-qa-2023/-/ tree/main/Task-B/data ↗
  16. A. Ramachandran, “Unveiling Google’s Gemini 2.0: A Comprehensive Study of its Multimodal AI Design, Advanced Architecture, and Real-World Applications,” ResearchGate, 2024. [Online]. Available: https://www.researchgate.net/publication/387089907 Unveiling Google% 27s Gemini 20 A Comprehensive Study of its Multimodal AI Design Advanced Architecture and Real-World Applications ↗
  17. M. Salama, The Qur’an and Modern Arabic Literary Criticism: From Taha to Nasr, Bloomsbury Publishing, 2018. find
  18. M. Zekiye, M. Bakouri, M. Oudah, and A. Mourad, “Al-Jawaab team: Generative constraint modeling with dense passage retrieval for Qur’anic question answering,” in Proc. 1st Workshop on Question Answering on the Quran, pp. 126–133, 2023. find
  19. Z. Zhang, E. Strubell, and E. Hovy, “Comparing span extraction methods for semantic role labeling,” in Proc. 5th Workshop on Structured Prediction for NLP (SPNLP), pp. 67–77, 2021. find
  20. X. Zhang, R. Mao, and E. Cambria, “A survey on syntactic processing techniques,” Artificial Intelligence Review, vol. 56, no. 6, pp. 5645–5728, 2023. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/basem-2025-few-shot-prompting-extractive-quranic/