Elamur Research⌕
☰
home / papers / 2025 / Tokenizers United at QIAS-2025: RAG-Enhanced Question Answering for Islamic Studies by Integrating Semantic Retrieval with Generative Reasoning

Tokenizers United at QIAS-2025: RAG-Enhanced Question Answering for Islamic Studies by Integrating Semantic Retrieval with Generative Reasoning

Paper · 2025 · Mohamed Samy, Mayar Boghdady, Marwan El Adawi et al.

Summary

The paper describes a system built for QIAS 2025 Sub Task 2, which tests how well models answer four-option multiple-choice questions about Islamic knowledge such as theology, jurisprudence, biography and ethics. The team argues that generic LLMs often hallucinate or lack grounding in this domain, so they built a retrieval-augmented pipeline. Classical reference books supplied by the organizers were cleaned (diacritics, Tatweel, URLs and similar noise removed), cut into overlapping chunks tagged with metadata, embedded with Muffakir_Embedding and stored in a Qdrant vector index. For each question, the top passages are placed into a prompt for Gemini 2.5 Flash Lite, which chooses the answer.

The authors compared several LLMs (Qwen3, Aya, Silma, Allam), embedding models, rerankers and chain-of-thought prompting. The best setup reached about 84% on the development set and 87% on the test set, placing 5th of 10 teams. Their main lesson is that the choice of embedding and the quality of retrieval mattered more than model size or added reasoning layers.

Key points

  • The best development configuration was Gemini 2.5 Flash Lite with Muffakir_Embedding and plain cosine retrieval of the top 10 chunks, scoring 84.29%. The same setup with a Flash reranker added dropped to 77.9%.
  • The final settings were 400-character chunks with a 100-character overlap, 768-dimensional vectors, roughly 15,000 vectors in Qdrant using HNSW, and Top-K = 10. This setup scored 87.00% on the test set and ranked 5th of 10.
  • Development accuracy varied widely across configurations, from 44.29% (Aya 8B with silma-embedding-matryoshka-v0.1) to 84.29%. Qwen3 results did not improve consistently with size, ranging from 54.43% to 78.00%.
  • Test accuracy fell as question difficulty rose: 89.14% on beginner questions, 83.43% on intermediate and 75.43% on advanced.
  • The knowledge base was drawn from organizer-provided classical works, including Al-Itqan fi Ulum al-Qur'an, Al-Sirah wa al-Shama'il, Tashnif al-Masamih and Manhaj al-Naqd fi Ulum al-Hadith. The authors report that BGE-reranker-v2-m3 and CoT gave little or no benefit compared with improving retrieval.

Interesting ideas

  • I1. The ablation changed one component at a time (generator, embedding, reranker, prompting) and found that the retrieval representation affected accuracy far more than the size of the generator, which shows where effort actually pays off.
  • I2. Adding a reranker, which seemed like an obvious improvement, actually reduced accuracy, so every extra layer should prove its value against the simpler baseline.
  • I3. Answers are grounded in a fixed, curated reference corpus, and each chunk carries an identifier, position and keywords, so retrieved evidence can be traced back to its source.
  • I4. Reporting accuracy by difficulty level reveals that the system is weaker on advanced reasoning, a gap that a single overall score would hide.
  • I5. The cleaning step strips orthographic noise (diacritics, elongation characters, URLs, emails) before indexing, so that surface variation does not split items that should match.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run per-stage ablations, for example swapping the parser or the frame extractor while keeping everything else fixed, to find which step limits end-to-end answer accuracy before scaling any model.
  • I2 (learn): Treat the plain pipeline as a mandatory control and accept a new repair rule or teacher tier only if it beats that control on held-out data.
  • I3 (world): Attach source IDs and offsets to compiled knowledge facts, such as WordNet or Wikidata entries, so every inference in the world of a text can cite the fact it relied on.
  • I4 (answer): Break school-course test scores down by grade or question type, separating recall from multi-step reasoning, to show which reader skills are missing.
  • I5 (tokens): Normalize typographic variants such as curly quotes, soft hyphens and URLs consistently before tagging, so that lemma and morphology lookups do not fail on surface noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nile UniversityGiza · Egypt14 works · 158 citations🎓Nile UniversityArua · Uganda4 works · 4 citations🎓Nile UniversityKhartoum · Sudan2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 4Research subjects and languages the work is about.Reasoning · 1,685 worksQuestion answering · 487 worksInformation retrieval & search · 552 worksTokenization & segmentation · 124 works

Cite

Cite as Mohamed Samy, Mayar Boghdady, Marwan El Adawi, Mohamed Nassar and Ensaf Hussein. 2025. Tokenizers United at QIAS-2025: RAG-Enhanced Question Answering for Islamic Studies by Integrating Semantic Retrieval with Generative Reasoning. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.133

@inproceedings{samy2025tokenizers,
  title = {{Tokenizers United at QIAS-2025: RAG-Enhanced Question Answering for Islamic Studies by Integrating Semantic Retrieval with Generative Reasoning}},
  author = {Mohamed Samy, Mayar Boghdady, Marwan {El Adawi}, Mohamed Nassar and Ensaf Hussein},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.133},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.133}
}

References

13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Al-Qurishi, Sarah AlQaseemi, and Riad Soussi. 2022. Aralegal-bert: A pretrained language model for arabic legal text. arXiv preprint arXiv:2210.08284. ↗
  2. Ahmet Yusuf Alan, Enis Karaarslan, and Ömer Aydin. 2024. A rag-based question answering system proposal for understanding islam: Mufassirqas llm. CoRR, abs/2401.15378. ↗
  3. Sarah Alnefaie, Eric Atwell, and Mohammad Ammar Alsalka. 2023. Islamic question answering systems survey and evaluation criteria. International Journal on Islamic Applications in Computer Science and Technology, 11(1):9–18. find
  4. Abdessalam Bouchekif, Samer Rashwani, Mohammed Ghaly, Mutaz Al-Khatib, Emad Mohamed, Wajdi Zaghouani, Heba Sbahi, Shahd Gaben, and Aiman Erbad. 2025a. Qias 2025: Overview of the shared task on islamic inheritance reasoning and knowledge assessment. In Proceedings of The Third Arabic Natural Language Processing Conference, ArabicNLP 2025, Suzhou, China, November 5–9, 2025. Association for Computational Linguistics.
  5. Abdessalam Bouchekif, Samer Rashwani, Heba Sbahi, Shahd Gaben, Mutaz Al-Khatib, and Mohammed Ghaly. 2025b. Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation. In Proceedings of The Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou, China. Association for Computational Linguistics.
  6. Romina Etezadi and Mehrnoush Shamsfard. 2021. A knowledge-based approach for answering complex questions in persian. CoRR, abs/2107.02040. ↗
  7. Arash Ghafouri, Hasan Naderi, Mohammad Aghajani Asl, and Mahdi Firouzmandi. 2023. Islam-icpcqa: A dataset for persian multi-hop complex question answering in islamic text resources. CoRR, abs/2304.11664. ↗
  8. Rana Malhas. 2023. Fine-tuning arabic qa models for qur’an qa task. In Proceedings of the 2022 Workshop on Open-Source Arabic Corpora and Tools (OSACT 2022). find
  9. Reham Mohamed, Maha Ragab, Heba Abdelnasser, Nagwa M. El-Makky, and Marwan Torki. 2015. Al-bayan: A knowledge-based system for arabic answer selection. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 226–230. Association for Computational Linguistics. ↗
  10. Damith Premasiri, Tharindu Ranasinghe, Wajdi Zaghouani, and Ruslan Mitkov. 2022. Dtw at qur’an qa 2022: Utilizing transfer learning with transformers for question answering in a low-resource domain. In Proceedings of the Qur’an QA 2022 Shared Task. find
  11. Faiza Qamar, Seemab Latif, and Rabia Latif. 2024. A benchmark dataset with larger context for non-factoid question answering over islamic text. arXiv preprint arXiv:2409.09844. ↗
  12. Muhammad Razif Rizqullah, Ayu Purwarianti, and Alham Fikri Aji. 2023. Qasina: Religious domain question answering using sirah nabawiyah. In arXiv preprint. find
  13. Kamyar Zeinalipour, Neda Jamshidi, Fahimeh Akbari, Marco Maggini, Monica Bianchini, and Marco Gori. 2025. Persianmcq-instruct: A comprehensive resource for generating multiple-choice questions in persian. In Proceedings of the First Workshop on Language Models for Low-Resource Languages (LoResLM), pages 344–372. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/samy-2025-tokenizers-united-qias-2025-rag/