Elamur Research⌕
☰
home / papers / 2022 / GOF at Qur'an QA 2022: Towards an Efficient Question Answering For The Holy Qu'ran In The Arabic Language Using Deep Learning-Based Approach

GOF at Qur'an QA 2022: Towards an Efficient Question Answering For The Holy Qu'ran In The Arabic Language Using Deep Learning-Based Approach

Paper · 2022 · Ali Mostafa and Omar Mohamed

Summary

The paper tackles extractive question answering over the Holy Qur'an. This is hard because the scripture is written in Classical Arabic, while most Arabic NLP resources and pre-trained models target Modern Standard Arabic. The authors entered the Qur'an QA 2022 shared task (OSACT5), which uses the small QRCD dataset. To compensate for QRCD's size, they chose AraELECTRA after comparing it with other Arabic encoders, and fine-tuned it in successive stages:

  1. First on Ar-TyDi QA.
  2. Then on the larger Arabic-SQuAD and ARCD collections.
  3. Finally on QRCD itself.

They also tried several training losses (cross-entropy, focal loss, focal loss with label smoothing, Dice loss). The full staged pipeline with plain cross-entropy performed best. It achieved a partial Reciprocal Rank (pRR) of 66.9% in development and 54.59% when evaluated on the test split. The work matters because it shows that borrowing supervision from general-domain reading-comprehension data can substantially lift a niche, low-resource QA task where in-domain labels are scarce.

Key points

  • QRCD data. QRCD contains 1,093 question–passage pairs and 1,337 question–passage–answer triplets, split 65/10/25 into train/dev/test (710/109/274 pairs). Passages repeat across questions, and the Qur'anic text comes from the Tanzil project in its simple-clean form.
  • Model. AraELECTRA is a 12-layer encoder with 136M parameters, 768 hidden size and a 512-token maximum input. It was pre-trained with replaced token detection on 8.8 billion words, mostly news.
  • Development-set baselines on QRCD alone. MARBERTv2 scored 50.46 pRR and AraELECTRA scored 56.07. Swapping the loss raised AraELECTRA to 57.60 with focal loss plus label smoothing and to 59.84 with focal loss.
  • Effect of extra fine-tuning stages.
    • Adding Ar-TyDi QA before QRCD gave 62.88.
    • Adding Arabic-SQuAD and ARCD as well gave 66.9 with cross-entropy.
    • In this full setup, focal loss (61.2) and Dice loss (61.4) did worse than cross-entropy.
  • Prior work. The authors' survey of earlier Qur'an QA found it was mostly ontology- and keyword-based. They argue the field lacked deep-learning experiments and reusable public gold test sets.

Interesting ideas

  • I1. Fine-tuning in a curriculum of intermediate datasets lets a model pick up the general skill of span extraction from plentiful data before it specialises on a tiny target set.
  • I2. The best loss changed with the regime: focal loss helped when only the small set was used but hurt once the model had more transferred knowledge. A tweak that wins in one data regime should therefore be re-tested in another.
  • I3. pRR, a ranking metric that gives partial credit to overlapping answer spans, rewards nearly correct answers instead of scoring all-or-nothing.
  • I4. The dataset deliberately pairs one passage with several questions and one question with several passages, which tests whether a system truly reads the passage rather than memorising it.
  • I5. Several pre-trained encoders were benchmarked on the target task before committing to one, rather than assuming the most popular choice would win.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Answer-span or role-labelling components could be pre-trained on large silver data first and then finished on small gold sets, with each stage evaluated separately to confirm it helps.
  • I2 (learn): Mova's training ablations should rerun loss or weighting choices after every change in data volume rather than keeping an earlier winner by default.
  • I3 (answer): QA on fables and tales could report a partial-overlap ranked score alongside exact match, separating "found the right region" from "got the exact span".
  • I4 (world): Test suites for the text world could ask several different questions about the same story, checking that one built world model supports them all consistently.
  • I5 (tree): When picking a backbone encoder for gparse, a quick fine-tune comparison on a small in-domain treebank slice could guide the choice before full training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Ali Mostafa and Omar Mohamed. 2022. GOF at Qur'an QA 2022: Towards an Efficient Question Answering For The Holy Qu'ran In The Arabic Language Using Deep Learning-Based Approach. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mostafa2022gof,
  title = {{GOF at Qur'an QA 2022: Towards an Efficient Question Answering For The Holy Qu'ran In The Arabic Language Using Deep Learning-Based Approach}},
  author = {Ali Mostafa and Omar Mohamed},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.12/}
}

References

44 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelmegied, A., Ayman, Y., Eid, A., El-Makky, N., Fathy, A., Khairy, G., Nagi, K., Nabil, M., and Yousri, M. (2017). A modified version of alquans: An arabic language question answering system. In International Joint Conference on Knowledge Discovery, Knowledge Engineering, and Knowledge Management, pages 184–199. Springer. find
  2. Abdelnasser, H., Ragab, M., Mohamed, R., Mohamed, A., Farouk, B., El-Makky, N. M., and Torki, M. (2014). Al-bayan: an arabic question answering system for the holy quran. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 57–64.
  3. Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2021). ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online, August. Association for Computational Linguistics. ↗
  4. Adany, M. A. H. et al. (2017). An automatic question answering system for the Arabic Quran. Ph.D. thesis, Sudan University of Science and Technology. find
  5. Ali, A. Y. (2000). The holy Qur’an. Wordsworth Editions. find
  6. Antoun, W., Baly, F., and Hajj, H. ). Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9.
  7. Antoun, W., Baly, F., and Hajj, H. (2020). Araelectra: Pre-training text discriminators for arabic language understanding. Arabic. ). Arabic language. ↗
  8. Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016). Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. ↗
  9. Brini, W., Ellouze, M., Mesfar, S., and Belguith, L. H. (2009). An arabic question-answering system for factoid questions. In 2009 International Conference on Natural Language Processing and Knowledge Engineering, pages 1–7. IEEE. find
  10. Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., and Palomaki, J. (2020a). Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. ↗
  11. Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. (2020b). Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. ↗
  12. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  13. Dice, L. R. (1945). Measures of the amount of ecologic association between species. Ecology, 26(3):297– 302. ↗
  14. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems, 27. find
  15. Hakkoum, A. and Raghay, S. (2016). Semantic q&a system on the qur’an. Arabian Journal for Science and Engineering, 41(12):5205–5214. ↗
  16. Hamdelsayed, M. A. and Atwell, E. (2016a). Using arabic numbers (singular, dual, and plurals) patterns to enhance question answering system results. In IMAN’2016 4th International Conference on Islamic Applications in Computer Science and Technologies. Leeds. find
  17. Hamdelsayed, M. A. and Atwell, E. (2016b). Islamic applications of automatic question-answering. find
  18. Hamdelsayed, M. A., Mohamed, E. M. E., Saeed, M. T. M., Ai, A. M., Mhmoud, E. B. E. M., Mahmoud, M. A., Shamat, A., and Atwell, E. (2017). Islamic application of question answering systems: Comparative study. Journal of Advanced Computer Science and Technology Research, 7(1):29–41. find
  19. Hamed, S. K. and Aziz, M. J. A. (2016). A question answering system on holy quran translation based on question expansion technique and neural network classification. J. Comput. Sci., 12:169–177. ↗
  20. Hamoud, B. and Atwell, E. (2016). Using an islamic question and answer knowledge base to answer questions about the holy quran. International Journal on Islamic Applications in Computer Science And Technology, 4(4):20–29. find
  21. Jang, Y., Lee, H., Hwang, S. J., and Shin, J. (2019). Learning what and where to transfer. In International Conference on Machine Learning, pages 3030–3039. PMLR. ↗
  22. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  23. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. ↗
  24. Li, X., Sun, X., Meng, Y., Liang, J., Wu, F., and Li, J. (2019). Dice loss for data-imbalanced nlp tasks. arXiv preprint arXiv:1911.02855. ↗
  25. Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988. ↗
  26. Malhas, R. and Elsayed, T. (2020). Ayatec: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6):1–21. find
  27. Malhas, R. and Elsayed, T. (2022). Official repository of qur’an qa shared task. https://gitlab.com/bigirqu/quranqa. ↗
  28. Malhas, R., Mansour, W., and Elsayed, T. (2022). find
  29. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  30. McCloskey, M. and Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier. ↗
  31. Menegola, A., Fornaciali, M., Pires, R., Bittencourt, F. V., Avila, S., and Valle, E. (2017). Knowledge transfer for melanoma screening with deep learning. In 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017), pages 297–300. IEEE. find
  32. Mohammed, F., Nasser, K., and Harb, H. M. (1993). A knowledge based arabic question answering system (aqas). ACM SIGART Bulletin, 4(4):21–30. ↗
  33. Mozannar, H., Maamary, E., El Hajal, K., and Hajj, H. (2019). Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy, August. Association for Computational Linguistics.
  34. Nabil, M., Abdelmegied, A., Ayman, Y., Fathy, A., Khairy, G., Yousri, M., El-Makky, N. M., and Nagi, K. (2017). Alquans-an arabic language question answering system. In KDIR, pages 144–154. find
  35. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. ↗
  36. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. ↗
  37. Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822. ↗
  38. Shmeisani, H., Tartir, S., Al-Na’ssaan, A., and Naji, M. (2014). Semantically answering questions from the holy quran. In International Conference on Islamic Applications in Computer Science And Technology, pages 1–8. find
  39. Silver, D. L., Yang, Q., and Li, L. (2013). Lifelong machine learning systems: Beyond learning algorithms. In 2013 AAAI spring symposium series. find
  40. Sorensen, T. A. (1948). A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on danish commons. Biol. Skar., 5:1–34. find
  41. Thrun, S. and Pratt, L. (2012). Learning to learn. Springer Science & Business Media. find
  42. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
  43. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2018). Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. ↗
  44. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mostafa-2022-gof-qur-qa-2022-efficient/