Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models

Summary

The paper addresses retrieving Qur'anic passages that answer questions posed in Modern Standard Arabic, when the passages themselves are in Classical Arabic. The training data from the Qur'an QA 2023 shared task was small, and earlier systems ranked passages poorly. The authors grew the 251 original questions to 629 by writing new ones and drawing on outside sources: the Quran QA 2022 data, a Kaggle set, a Tafseer Q&A book, Hugging Face Tafseer and parallel-text corpora, and a list of plants named in the Qur'an. They then paraphrased every question twice, producing 1895 items labelled as single-answer, multi-answer or zero-answer. After removing duplicates and normalising text, they fine-tuned several encoders on the enlarged set: AraBERT-base and -large, CAMeLBERT, AraELECTRA, RoBERTa and a SQuAD-tuned BERT. The test and development splits were left unchanged. AraBERT-base gave the strongest ranking scores, raising MAP@10 from 0.22 to 0.36 and MRR from 0.37 to 0.59. Handling of unanswerable questions also improved. The study argues that, for a niche low-resource domain, curating and diversifying the data can matter as much as the choice of model.

Key points

The dataset grew from 251 to 629 questions, then to 1895 after two paraphrases per question. The 52 test and 25 development questions stayed fixed.
With AraBERT-base, MAP@10 rose from 0.22 to 0.36 and MRR from 0.37 to 0.59, which the authors report as gains of about 63% and 59%.
For bert-squad-accelerate, no-answer recall and no-answer precision both went from 0.25 to 0.75. No-answer scores in the tables only take values in steps of 0.25.
The other models improved as follows:
- AraELECTRA-base: Recall@15 rose from 0.42 to 0.57.
- RoBERTa: gains were small, with MAP@10 rising from 0.05 to 0.12.
- AraBERT-large: MAP@10 stayed at 0.28.
The paper is inconsistent about its best model. The contributions credit AraBERT-large, while the tables and abstract favour AraBERT-base. The conclusion also mentions ensembling and thresholding without describing them in detail.

Interesting ideas

I1. Paraphrasing each training question several times while keeping its gold passages gives cheap lexical variety for a tiny QA set; the evaluation split must stay untouched. — For Mova (learn): We could expand small question sets for fables and school tests by rewording them, then check that the frames and answers stay identical across variants.
I2. Labelling every question up front as single-answer, multi-answer or zero-answer turns answer cardinality into an explicit target rather than an afterthought. — For Mova (answer): Our readers could predict whether zero, one or several answers are expected before searching the world model, and report this choice as part of the explanation.
I3. Unanswerable questions get their own recall and precision, so a system cannot hide weak abstention behind good ranking scores. — For Mova (learn): Evaluation with negative controls could report a separate "correctly says no answer" score for questions the text world cannot support.
I4. The task bridges a register gap: queries are in a modern variety, while the source text is in a classical one, and both sides are normalised. — For Mova (tokens): Archaic forms in old tales (thou, hath, ere) could be mapped to modern lemmas in the morphology layer, so later stages see one consistent vocabulary.
I5. Reporting the same metrics before and after data expansion, across many backbones, shows how much of the gain comes from data rather than from the model. — For Mova (learn): When adding silver data, we could retrain several parser variants on old and new data side by side, to attribute improvements honestly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining · Question answering · Semitic languages · Information retrieval & search

Institutions

🎓October University of Modern Sciences and Arts

Cairo · Egypt7 works · 25 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.11431 · arXiv 2412.11431

@misc{hamdi2024optimized,
  title = {{Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models}},
  author = {Hamdi;, Mohamed Basem; Islam Oshallah; Baraa Hikal; Ali and Mohamed, Ammar},
  year = {2024},
  doi = {10.48550/arxiv.2412.11431},
  eprint = {2412.11431},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.11431}
}

References

5 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alawwad, H., Alawwad, L., Alharbi, J., Alharbi, A.: Ahjl at qur’an qa 2023 shared task: Enhancing passage retrieval using sentence transformer and translation. In: Proceedings of ArabicNLP 2023, pp. 702–707 (2023) 2. Aljamel, A., Khalil, H., Aburawi, Y.: Comparative study of fine-tuned bert-based models and rnn-based models. case study: Arabic fake news detection. The International Journal of Engineering and Information Technology (IJEIT) 12(1), 56–64 (2024) 3. Antoun, W., Baly, F., Hajj, H.: Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104 (2020). URL https://arxiv.org/abs/2003.00104 4. Antoun, W., Baly, F., Hajj, H.: Araelectra: Pre-training text discriminators for arabic language understanding. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 191–195. Association for Computational Linguistics (2021). URL https://aclanthology.org/2021.wanlp-1.21/ 5. Ashor, Q.: 1000 QAs from the Holy Qur’an. ↗ our card
Noor Book (2023). URL https://quranpedia.net/book/451/1/259 6. Clark, J., et al.: Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the ACL (2020) 7. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186. Association for Computational Linguistics (2019). URL https://aclanthology.org/N19-1423/ 8. Elkomy, M.: Quran qa 2022 dataset (2022). GitHub Repository 9. Elkomy, M., Sarhan, A.: Tce at qur’an qa 2023 shared task: Low resource enhanced transformer-based ensemble approach for qur’anic qa. In: Proceedings of ArabicNLP 2023, pp. 728–742. Association for Computational Linguistics, Singapore (Hybrid) (2023) 10. Essam, M., Deif, M., Elgohary, R.: Deciphering arabic question: A dedicated survey on arabic question analysis methods, challenges, limitations and future pathways. Artificial Intelligence Review 57(9), 1–37 (2024) 11. GARDEN, Q.B.: List of plants citation in quran and hadith v5.pdf (2024) 12. Hamdi, A., Shaban, K., Zainal, A.: A review on challenging issues in arabic sentiment analysis. Journal of Computer Science (2016) 13. Hamdi, A., Shaban, K., Zainal, A.: Clasenti: a class-specific sentiment analysis framework. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 17(4), 1–28 (2018) 14. Hillman, J., Baydoun, E.: Quality assurance and relevance in academia: a review. ↗ our card
Springer (2019) 15. ImruQays: Quran-classical-arabic-english parallel texts dataset on hugging face (2024). URL https://huggingface.co/datasets/ImruQays/ Quran-Classical-Arabic-English-Parallel-texts Authors Suppressed Due to Excessive Length 16. Inoue, G., Habash, N.: Camelbert: A language model for arabic. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 270–278. Association for Computational Linguistics (2021). URL https://aclanthology.org/2021.wanlp-1.29/ 17. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). URL https://arxiv.org/abs/1907.11692 18. Liu, Z., Li, Y., Chen, N., Wang, Q., Hooi, B., He, B.: A survey of imbalanced learning on graphs: Problems, techniques, and future directions. arXiv preprint arXiv:2308.13821 (2023) 19. Mahmoudi, G., Eetemadi, S., Morshedzadeh, Y.: A multi-task transfer learning approach for qur’an-related question answering. In: Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023). ↗
ACL Anthology (2023) 20. Malhas, M., et al.: Qur’an qa 2023 shared task: Overview of passage retrieval and reading comprehension tasks over the holy qur’an. In: ArabicNLP-WS 2023, pp. 1–13. Association for Computational Linguistics (2023) 21. Malhas, R.: Arabic question answering on the holy qur’an. Ph.D. thesis, Ph.D. thesis (2023) 22. Mobassir: Quran qa dataset on kaggle (2024). URL https://www.kaggle.com/datasets/mobassir/quranqa/code 23. Qamar, F., Latif, S., Latif, R.: A benchmark dataset with larger context for non-factoid question-answering over islamic text. Preprint submitted to Elsevier (2024) 24. Rashad, M.: Quran-tafseerbook dataset on hugging face (2024) 25. Sardar, Z.: Reading the Qur’an: The contemporary relevance of the sacred text of Islam. ↗
Oxford University Press (2017) 26. Sun, L., Xia, C., Yin, W., Liang, T., Yu, P.S., He, L.: Mixup-transformer: Dynamic data augmentation for nlp tasks. arXiv preprint arXiv:2010.02394 (2020) 27. Zheng, H., Shen, L., Tang, A., Luo, Y., Hu, H., Du, B., Tao, D.: Learn from model beyond fine-tuning: A survey. arXiv preprint arXiv:2310.08184 (2023) ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/basem-2024-optimized-quran-passage-retrieval-using/

📄 Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models