Elamur Research⌕
☰
home / papers / 2022 / Stars at Qur'an QA 2022: Building Automatic Extractive Question Answering Systems for the Holy Qur'an with Transformer Models and Releasing a New Dataset

Stars at Qur'an QA 2022: Building Automatic Extractive Question Answering Systems for the Holy Qur'an with Transformer Models and Releasing a New Dataset

Paper · 2022 · Ahmed Sleem, Eman Mohammed lotfy Elrefai, Marwa Mohammed Matar et al.

Summary

The paper describes a team's entry to the Qur'an QA 2022 shared task (OSACT5 @ LREC 2022). In this task a system receives a Modern Standard Arabic question and a short Classical Arabic Qur'anic passage, and it must extract the answer span. The official QRCD training set is small and repeats many questions across passages. To address this, the authors built a supplementary dataset, AQAQ. They took questions from the AQQAC corpus, whose answers mostly come from al-Tabari's tafsir rather than the Qur'an itself, and manually re-anchored them to Qur'anic passages in QRCD format. They compared three transformer-based systems built on araBERT:

  • fine-tuning on the expanded data;
  • a k-fold ensemble of the strongest models;
  • Qur'anBERT, which is araBERT further trained with masked language modelling on raw Qur'an verses before QA fine-tuning.

The best system, trained on expanded AQAQ, reached 0.528 pRR on the test set and about 0.62 on the development set. All three approaches scored close to each other. The work matters because it shows that small, carefully curated in-domain data can move results noticeably in a very low-resource, high-stakes domain.

Key points

  • QRCD has 1,093 question–passage pairs and 1,337 question–passage–answer triplets, split into 710 train, 109 dev and 352 test samples. Answers are ranked spans that must appear in the passage.
  • AQAQ grew from about 625 to 732 samples and roughly doubled the training data. About 500 questions came from AQQAC (1,224 items). An automated conversion was abandoned in favour of manual annotation.
  • Training on QRCD alone gave pRR of 44% or lower. Adding the team's own data raised pRR by about 10 points.
  • Arabic SQuAD (48,344 questions, machine-translated from SQuAD) helped only sometimes. Pretraining on it alone first underperformed, while mixing random portions of it with Qur'anic data occasionally helped.
  • Test results:
SystempRREMF1@1
Expanding AQAQ0.5280.2560.507
K-folds ensemble0.521——
Qur'anBERT0.502——

Code and data are released on GitHub.

Interesting ideas

  • I1. An existing question bank whose answers point to the wrong source (commentary) can be salvaged: keep its questions and hand-relink each answer to a span in the target text, which yields extractive QA data cheaply.
  • I2. Continuing masked-LM training on the bare target-domain text before task fine-tuning adapts a general model to an archaic or unusual register.
  • I3. Shuffling random-sized slices of out-of-register data into in-domain training worked better than a separate pretraining stage on that data.
  • I4. Scoring with a ranked, partial-credit metric (pRR) accepts several valid answers per question in order of quality.
  • I5. Training models on different k-fold splits and ensembling the best ones stabilises predictions when data is tiny.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Mova could reuse question lists from school tests and re-anchor their answers to exact spans in the fable or tale text, so each answer stays traceable to the passage.
  • I2 (tree): gparse could be briefly domain-adapted on raw fairy-tale or old-style English text before parser training, to cope with archaic word forms.
  • I3 (learn): when silver data from modern corpora is combined with small gold sets of literary text, Mova's training should test interleaved proportions rather than sequential stages.
  • I4 (answer): the QA evaluation on fables could accept ranked alternative answer spans and give partial credit, instead of strict exact match.
  • I5 (learn): the same fold-based diversity could create extra witness parsers whose agreement filters silver data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Ahmed Sleem, Eman Mohammed lotfy Elrefai, Marwa Mohammed Matar and Haq Nawaz. 2022. Stars at Qur'an QA 2022: Building Automatic Extractive Question Answering Systems for the Holy Qur'an with Transformer Models and Releasing a New Dataset. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{sleem2022stars,
  title = {{Stars at Qur'an QA 2022: Building Automatic Extractive Question Answering Systems for the Holy Qur'an with Transformer Models and Releasing a New Dataset}},
  author = {Ahmed Sleem, Eman Mohammed lotfy Elrefai, Marwa Mohammed Matar and Haq Nawaz},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.18/}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelnasser, H., Ragab, M., Mohamed, R., Mohamed, A., Farouk, B., El-Makky, N. M., and Torki, M. (2014). Al-bayan: an arabic question answering system for the holy quran. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 57–64.
  2. Alqahtani, M. and Atwell, E. (2018). Annotated corpus of arabic al-quran question and answer. find
  3. Alsubhi, K., Jamal, A., and Alhothali, A. (2021). Pre-trained transformer-based approach for arabic question answering: A comparative study. arXiv preprint arXiv:2111.05671. ↗
  4. Antoun, W., Baly, F., and Hajj, H. (2020a). Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  5. Antoun, W., Baly, F., and Hajj, H. (2020b). Araelectra: Pre-training text discriminators for arabic language understanding. arXiv preprint arXiv:2012.15516. ↗
  6. Bakari, W., Bellot, P., and Neji, M. (2016). Literature review of arabic question-answering: Modeling, generation, experimentation and performance analysis. In Flexible Query Answering Systems 2015, pages 321–334, cham. Springer International Publishing. ↗
  7. Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. (2020). Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. ↗
  8. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  9. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  10. Elhindi, Y. (2017). Metaphors in the quran: A thematic categorization. QURANICA-International Journal of Quranic Research, 9(1):1–20. ↗
  11. Hu, D. (2019). An introductory survey on attention mechanisms in nlp problems. In Proceedings of SAI Intelligent Systems Conference, pages 432–448. Springer. 4 https://github.com/EmanElrefai/Qur’ an_QA Khaled, S., Siddiqui, S., Alkhatib, M., and Monem, A. A. (2018). Challenges in arabic natural language processing. Computational Linguistics, Speech and Image Processing for Arabic Language, pages 59— -83. ↗
  12. Khorami, M. (2014). Eloquence of repetition in quran and arabic old poetry. Language Related Research, 5(2):91–110. find
  13. Malhas, R. and Elsayed, T. (2020). Ayatec: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6):1–21. find
  14. Malhas, R., Mansour, W., and Elsayed, T. (2022). find
  15. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  16. Marcus, M., Kim, G., Marcinkiewicz, M. A., Mac-Intyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. (1994). The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
  17. Mishra, A. and Jain, S. K. (2016). A survey on question answering systems with classification. Journal of King Saud University-Computer and Information Sciences, 28(3):345–361. ↗
  18. Mozannar, H., Hajal, K. E., Maamary, E., and Hajj, H. (2019). Neural arabic question answering. arXiv preprint arXiv:1906.05394. pritesh1. (2022). An explanatory guide to bert tokenizer. ↗
  19. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250. ↗
  20. Tahani, A., Azmi, A. M., Aboalsamh, H. A., Cambria, E., and Hussain, A. (2021). Arabic question answering system: A survey. Artificial Intelligence Review. find
  21. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sleem-2022-stars-qur-qa-2022-building/