Elamur Research⌕
☰
home / papers / 2023 / Qur'an QA 2023 Shared Task: Overview of Passage Retrieval and Reading Comprehension Tasks over the Holy Qur'an

Qur'an QA 2023 Shared Task: Overview of Passage Retrieval and Reading Comprehension Tasks over the Holy Qur'an

Paper · 2023 · Rana Malhas, Watheq Mansour and Tamer Elsayed

Summary

The paper reports on the second edition of the Qur'an QA shared task, held at ArabicNLP 2023. It targets question answering over the Holy Qur'an, which is written in Classical Arabic while the questions are posed in Modern Standard Arabic (MSA). The organizers defined two sub-tasks. In passage retrieval (PR), a system ranks up to 10 answer-bearing passages from the Qur'anic Passage Collection (QPC). In machine reading comprehension (MRC), a system extracts every answer span from a given passage. Both tasks include questions that the Qur'an does not answer, and MRC also contains questions with several answers. The PR data, AyaTECv1.2, extends the earlier AyaTEC collection with 52 new test questions annotated by Qur'an specialists. Nine teams sent 22 PR runs and six teams sent 17 MRC runs. Encoder models in the BERT family outperformed generative LLMs. The winners used self-ensembles of fine-tuned AraELECTRA and AraBERT. The task gives Arabic NLP a shared benchmark for a sensitive, low-resource domain where reliable QA is in demand.

Key points

  • The QPC was built by splitting all 114 Surahs into 1,266 topical passages, following the printed Thematic Holy Qur'an.
  • AyaTECv1.2 has 251 questions: 199 kept from AyaTECv1.1 plus 52 new ones. Of these, 37 (about 15%) have no answer in the Qur'an. The splits are 174/25/52 questions and 972/160/427 question-passage pairs.
  • A passage counts as relevant if it contains any gold verse-based answer, either fully or partially. These relevance judgments were generated automatically.
  • The 2022 edition scored MRC with partial Reciprocal Rank (pRR). The 2023 MRC task uses partial Average Precision (pAP), which gives credit for partially matching spans.
  • The top MRC team (TCE) handled multi-answer questions with Maximum Marginal Likelihood (MML) fine-tuning. The runner-up (Al-Jawaab) found that GPT-4 sometimes answered outside the passage or leaked prompt text into its output, and needed post-processing heuristics to fix this.

Interesting ideas

  • I1. About 15% of the questions have no answer at all, so a system is rewarded for returning nothing, and the best system did this by thresholding its predicted likelihood of an answer.
  • I2. Fine-grained gold annotations at the verse level were automatically turned into coarser passage-level relevance by checking overlap, which avoided a second round of manual labeling.
  • I3. Partial-match metrics such as pAP and pRR give graded credit when a predicted span only overlaps the gold answer, instead of scoring it as a plain miss.
  • I4. MML fine-tuning lets probability mass spread over several valid answer spans, so the model is not pushed to commit overconfidently to a single one.
  • I5. Even with careful prompting, GPT-4 answers sometimes fell outside the passage or echoed the prompt, so the outputs had to be checked and repaired after generation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Fable and school-test readers could get unanswerable control questions and a confidence cutoff, so that "the text does not say" becomes a scored outcome rather than a forced guess.
  • I2 (learn): Evaluation sets annotated at the span or frame level could be projected the same way to sentence- or paragraph-level judgments, reusing one careful annotation for several granularities.
  • I3 (answer): Scoring extractive answers on tale QA by token overlap would separate near-misses from real failures and show where the reader's span boundaries go wrong.
  • I4 (learn): When several witness parsers or teachers give different but acceptable outputs, training against the whole set of acceptable answers could keep the model from overfitting to one arbitrary choice.
  • I5 (learn): Outputs from the LLM teacher tiers should be verified mechanically, for example by confirming that an answer is a real substring of the source, before they enter silver data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations🎓The University of QueenslandBrisbane · Australia14 works · 4,603 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 worksInformation retrieval & search · 473 works

Cite

Cite as Rana Malhas, Watheq Mansour and Tamer Elsayed. 2023. Qur'an QA 2023 Shared Task: Overview of Passage Retrieval and Reading Comprehension Tasks over the Holy Qur'an. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.76

@inproceedings{malhas2023qur,
  title = {{Qur'an QA 2023 Shared Task: Overview of Passage Retrieval and Reading Comprehension Tasks over the Holy Qur'an}},
  author = {Rana Malhas, Watheq Mansour and Tamer Elsayed},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.76},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.76}
}

References

44 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Basem H. Ahmed, Motaz K. Saad, and Eshrag A. Refaee. 2022. QQATeam at Quran QA 2022: Fine-Tunning Arabic QA Models for Quran QA Task. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  2. Zeyad Ahmed. 2023. Arabic squad v2.0 dataset based on the popular squadv2.0 with unanswered questions for more challenging task. https://huggingface.co/datasets/ ZeyadAhmed/Arabic-SQuADv2.0. Accessed: September 28, 2023. ↗
  3. Hend Al-Khalifa, Tamer Elsayed, Hamdy Mubarak, Abdulmohsen Al-Thubaity, Walid Magdy, and Kareem Darwish. 2022. Proceedinsg of the 5th workshop on osact with shared tasks on qur’an qa and fine-grained hate speech detection. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection. find
  4. Hessa A. Alawwad, Lujain A. Alawwad, Jamilah Alharbi, and Abdullah I. Alharbi. 2023. AHJL at Qur’an QA 2023 Shared Task: Enhancing Passage Retrieval using Sentence Transformer and Translation. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore.
  5. Sarah Alnefaie, Abdullah N. Alsaleh, Eric Atwell, Mohammad Ammar Alsalka, and Abdulrahman Altah-han. 2023. LKAU23 at Qur’an QA 2023: Using Transformer Models for Retrieving Passages and Finding Answers to Questions from the Qur’an. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore.
  6. Mohammad Alqahtani and Eric Atwell. 2018. Annotated corpus of Arabic al-quran question and answer. find
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  9. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  10. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906. ↗
  11. Jimmy Lin and Boris Katz. 2006. Building a reusable test collection for question answering. Journal of the American Society for Information Science and Technology, 57(7):851–861. ↗
  12. Ghazaleh Mahmoudi and Yeganeh Morshedzadeh. 2023. GYM at Qur’an QA 2023 Shared Task: Multi-Task Transfer Learning for Quranic Passage Retrieval and Question Answering with Large Language Models. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. ↗
  13. Rana Malhas. 2023. Arabic Question Answering on the Holy Qur’an. Ph.D. thesis, Qatar University. ↗
  14. Muhammad Huzaifa Bashir, Aqil M Azmi, Haq Nawaz, Wajdi Zaghouani, Mona Diab, Ala Al-Fuqaha, and Junaid Qadir. 2022. Arabic natural language processing for Qur’anic research: a systematic review. Artificial Intelligence Review, pages 1–54. ↗
  15. Rana Malhas and Tamer Elsayed. 2020. AyaTEC: Building a Reusable Verse-based Test Collection for Arabic Question Answering on the Holy Qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing, 19(6). find
  16. Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Ph.D. thesis, Stanford University. find
  17. Rana Malhas and Tamer Elsayed. 2022. Arabic Machine Reading Comprehension on the Holy Qur’an using CL-AraBERT. Information Processing & Management, 59(6):103068. ↗
  18. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. ↗
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), page 4171–4186. Association for Computational Linguistics.
  20. Mohammed Alaa Elkomy and Amany Sarhan. 2023. TCE at Qur’an QA 2023 Shared Task: Low Resource Enhanced Transformer-based Ensemble Approach for Qur’anic QA. In Proceedings of the First Arabic Natural Language Processing Conference (Arabic-NLP 2023), Singapore.
  21. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821. ↗
  22. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. find
  23. Qur’an QA 2022: Overview of the First Shared Task on Question Answering over the Holy Qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022), pages 79–87.
  24. Aly Mostafa and Omar Mohamed. 2022. GOF at Qur’an QA 2022: Towards an Efficient Question Answering For The Holy Qu’ran In The Arabic Language Using Deep Learning-Based Approach. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  25. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, page 108–118. Association for Computational Linguistics.
  26. OpenAI. 2023a. GPT-4 Technical Report. Technical report. ↗
  27. OpenAI. 2023b. New and improved embedding model. https://openai.com/blog/new-and-improved-embedding-model. Accessed: September 27, 2023. ↗
  28. OpenAI. 2023c. OpenAI Embeddings API. https://platform.openai.com/docs/guides/ embeddings/. Accessed: September 27, 2023. ↗
  29. Adrien Pavao, Isabelle Guyon, Anne-Catherine Letournel, Dinh-Tuan Tran, Xavier Baro, Hugo Jair Escalante, Sergio Escalera, Tyler Thomas, and Zhen Xu. 2023. Codalab competitions: An open source platform to organize scientific challenges. Journal of Machine Learning Research, 24(198):1–6. find
  30. Damith Premasiri, Tharindu Ranasinghe, Wajdi Zaghouani, Ruslan Mitkov, Jamal Berrich, and Toumi Bouchentouf. 2022. DTW at Qur’an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain . In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  31. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822. ↗
  32. Nils Reimers. 2023. SBERT Semantic Search. https://www.sbert.net/examples/applications/semantic-search/README.html. Accessed: September 27, 2023. ↗
  33. Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. TSDAE: Using transformer-based sequential denoising auto-encoderfor unsupervised sentence embedding learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 671–688, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  34. Ahmed Wasfey, Eman Elrefai, Muhammad Marwa, and Nawaz Haq. 2022. Stars at Qur’an QA 2022: Building Automatic Extractive Question Answering Systems for the Holy Qur’an with Transformer Models and Releasing a New Dataset. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  35. Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. Pretrained transformers for text ranking: BERT and beyond. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials, pages 1–4, Online. Association for Computational Linguistics. find
  36. Abdulrezzak Zekiye and Fadi Amroush. 2023. Al-jawaab at Qur’an QA 2023 shared task: Exploring Embeddings and GPT Models for Passage Retrieval and Reading Comprehension. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore.
  37. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. ↗
  38. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 127– 137, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  39. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059.
  40. Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774. ↗
  41. Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. 2020. Beyond [cls] through ranking by generation. arXiv preprint arXiv:2010.03073. ↗
  42. A Maximilian Schreiner. 2023. Gpt-4 architecture, datasets, costs and more leaked. https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/. Accessed: September 28, 2023. ↗
  43. Marwan N. Swar. 2007. Mushaf Al-Tafseel Al-Mawdoo’ee. Dar Al-Fajr Al-Islami, Damascus. find
  44. Hariram Veeramani and Kaushik Roy. 2023. LowResContextQA at Qur’an QA 2023 Shared Task: Temporal and Sequential Representation Augmented Question Answering Span Detection. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/malhas-2023-qur-qa-2023-shared-task/