Elamur Research⌕
☰
home / papers / 2025 / Transformer Tafsir at QIAS 2025 Shared Task: Hybrid Retrieval-Augmented Generation for Islamic Knowledge Question Answering

Transformer Tafsir at QIAS 2025 Shared Task: Hybrid Retrieval-Augmented Generation for Islamic Knowledge Question Answering

Paper · 2025 · Muhammad Abu Ahmad, Mohamad Ballout, Raia Abu Ahmad et al.

Summary

The paper describes a system for the QIAS 2025 shared task, which tests how well LLMs answer multiple-choice questions on Islamic knowledge in Arabic. Subtask 1 covers inheritance law ('Ilm al-Mawārīth) and Subtask 2 covers general Islamic topics. A central difficulty is the gap between Modern Standard Arabic questions and Classical Arabic source texts.

The authors built a hybrid RAG pipeline without any fine-tuning. Knowledge bases came from the organizers' materials: IslamWeb fatwas for Subtask 1 and a corpus of classical books for Subtask 2. Retrieval runs as a funnel. BM25 works over heavily preprocessed text, a dense retriever (Arabic-Triplet-Matryoshka-V2) works over lightly preprocessed text, and a multilingual miniLMv2 cross-encoder reranks the results. The top passages go into a few-shot prompt with an expert-scholar persona, which is sent to Fanar or Mistral.

Retrieval lifted accuracy in every configuration. The gains were large on general knowledge, up to 25 points, but small on inheritance questions, which mostly require fractional arithmetic. The work shows both the value and the limits of plain retrieval for reasoning-heavy domains.

Key points

  • Data:

    • Subtask 1 has 9,450 train, 1,500 validation and 1,000 test MCQs, plus 32,000 IslamWeb fatwas.
    • Subtask 2 has 800 validation and 1,000 test MCQs, drawn from 25 classical books.
    • Each fatwa is one retrieval unit. Books are split into overlapping paragraph chunks.

  • Retrieval funnel:

    • BM25 keeps the top 1000 candidates.
    • Arabic-Triplet-Matryoshka-V2 narrows these to 200 by cosine similarity.
    • The cross-encoder mmarco-mMiniLMv2-L12-H384-v1 picks the final 5 for the prompt.

  • Preprocessing: CAMeL Tools handles normalization, dediacritization and tokenization, and NLTK stopword lists are used. Stopword removal and tokenization apply only to the BM25 view.

  • Test accuracy (baseline → +RAG):

    ModelSubtask 1Subtask 2
    Fanar44% → 45%55% → 80%
    Mistral (mistral-saba-24b)35% → 39%69% → 79%

  • By difficulty, best Fanar system (Table 3):

    • Subtask 1 falls from 53.40% on beginner to 36.00% on advanced questions.
    • Subtask 2 falls from 81.86% to 73.33%.
    • The body text reports slightly different figures for these levels.

Interesting ideas

  • I1. The authors keep two normalizations of each chunk: an aggressive one for lexical matching and a light, meaning-preserving one for neural models. Both are linked to the original text, so any stage can recover the source span.
  • I2. Candidates are narrowed in stages, from a wide cheap lexical filter to semantic similarity to an expensive pairwise scorer. This keeps cost low while precision rises at each step.
  • I3. The retrieval unit is chosen per source: a whole fatwa stays intact for coherence, while long books are split into overlapping paragraphs so facts are not lost at boundaries.
  • I4. The error analysis separates the questions retrieval helps (factual recall) from those it cannot fix (fractional, multi-step inheritance arithmetic). The authors conclude that a symbolic reasoning module is needed.
  • I5. Every model is scored both with and without retrieval under one fixed configuration. The domain-specialized model gained less relatively than the general model, which hints at overlap with its pretraining data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the en crate could emit a heavy-normalized and a light-normalized view of each token, both with offsets into the raw text, so lemmas and matching never lose traceability to the original span.
  • I2 (answer): fable and tale QA readers could locate supporting sentences the same way, first by lemma overlap, then by frame similarity, then by a full tree-to-question alignment over only a handful of candidates.
  • I3 (world): when compiling knowledge from Wikidata or longer texts, the unit for building a text world could be chosen by document structure, with overlap so entities and events spanning a boundary are not dropped.
  • I4 (answer): maths word problems should go to an explicit symbolic solver over the world state, not to evidence lookup, and the question type could decide the routing.
  • I5 (learn): each new component could be evaluated against an ablated control on the same items, with gains read in light of what the model or teacher already knows.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Osnabrück UniversityOsnabrück · Germany7 works · 250 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 3,066 worksQuestion answering · 487 worksInformation retrieval & search · 552 works

Cite

Cite as Muhammad Abu Ahmad, Mohamad Ballout, Raia Abu Ahmad and Elia Bruni. 2025. Transformer Tafsir at QIAS 2025 Shared Task: Hybrid Retrieval-Augmented Generation for Islamic Knowledge Question Answering. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.124

@inproceedings{ahmad2025transformer,
  title = {{Transformer Tafsir at QIAS 2025 Shared Task: Hybrid Retrieval-Augmented Generation for Islamic Knowledge Question Answering}},
  author = {Muhammad Abu Ahmad, Mohamad Ballout, Raia Abu Ahmad and Elia Bruni},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.124},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.124}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. Arabicaqa: A comprehensive dataset for arabic question answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2049–2059.
  2. Raghad Al-Rasheed, Abdullah Al Muaddi, Hawra Aljasim, Rawan Al-Matham, Muneera Alhoshan, Asma Al Wazrah, and Abdulrahman AlOsaimy. 2025. Evaluating RAG pipelines for Arabic lexical information retrieval: A comparative study of embedding and generation models. In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 155–164, Abu Dhabi, UAE. Association for Computational Linguistics.
  3. Jumana Alsubhi, Mohammad D Alahmadi, Ahmed Alhusayni, Ibrahim Aldailami, Israa Hamdine, Ahmad Shabana, Yazeed Iskandar, and Suhayb Khayyat. 2025. Optimizing rag pipelines for arabic: A systematic analysis of core components. arXiv preprint arXiv:2506.06339. ↗
  4. Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz. 2024. A survey on rag with llms. Procedia computer science, 246:3781–3790. find
  5. Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics.
  6. Abdessalam Bouchekif, Samer Rashwani, Mohammed Ghaly, Mutaz Al-Khatib, Emad Mohamed, Wajdi Zaghouani, Heba Sbahi, Shahd Gaben, and Aiman Erbad. 2025a. Qias 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment. In Proceedings of The Third Arabic Natural Language Processing Conference, ArabicNLP 2025, Suzhou, China, November 5–9. Association for Computational Linguistics.
  7. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al-Khatib, and Mohammed Ghaly. 2025b. Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation. In Proceedings of The Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou, China. Association for Computational Linguistics.
  8. Hao Cheng, Hao Fang, Xiaodong Liu, and Jianfeng Gao. 2023. Task-aware specialization for efficient and robust dense retrieval for open-domain question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1864–1875, Toronto, Canada. Association for Computational Linguistics. find
  9. Samhaa R El-Beltagy and Mohamed A Abdallah. 2024. Exploring retrieval augmented generation in arabic. Procedia Computer Science, 244:296–307.
  10. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, and 67 others. 2025. Mmteb: Massive multilingual text embedding benchmark. arXiv preprint arXiv:2502.13595. ↗
  11. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1). ↗
  12. Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Morgan & Claypool Publishers. ↗
  13. Chip Huyen. 2024. AI Engineering: Building Applications with Foundation Models. O’Reilly Media, Incorporated. find
  14. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769–6781.
  15. Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, and 1 others. 2025. A survey of frontiers in llm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arXiv:2504.09037. ↗
  16. Zahra Khalila, Arbi Haza Nasution, Winda Monika, Aytug Onan, Yohei Murakami, Yasir Bin Ismail Radi, and Noor Mohammad Osmani. 2025. Investigating retrieval-augmented generation in quranic studies: A study of 13 open-source large language models. arXiv preprint arXiv:2503.16581. Cosine similarity; Top candidates = 200; Batch size = 8 for embedding computation Cross-Encoder Reranking: Model: Arabic BERT-based reranker; Final candidates = 5; Temperature = 0.1 for stable rankings Omer Nacar, Anis Koubaa, Serry Sibaee, Yasser Al-Habashi, Adel Ammar, and Wadii Boulila. 2025. Gate: General arabic text embedding for enhanced semantic textual similarity with matryoshka representation learning and hybrid loss training. arXiv preprint arXiv:2505.24581. A.2 ↗
  17. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  18. Islam Oshallah, Mohamed Basem, Ali Hamdi, and Ammar Mohammed. 2025. Cross-language approach for quranic qa. arXiv preprint arXiv:2501.17449. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ahmad-2025-transformer-tafsir-qias-2025-shared/