Elamur Research⌕
☰
home / papers / 2023 / LKAU23 at Qur'an QA 2023: Using Transformer Models for Retrieving Passages and Finding Answers to Questions from the Qur'an

LKAU23 at Qur'an QA 2023: Using Transformer Models for Retrieving Passages and Finding Answers to Questions from the Qur'an

Paper · 2023 · Sarah Alnefaie, Abdullah Alsaleh, Eric Atwell et al.

Summary

The paper describes the LKAU23 team's systems for the two subtasks of the Qur'an QA 2023 shared task: passage retrieval (PR) over the Qur'an and extractive machine reading comprehension (MRC) in Classical Arabic. Some questions in both subtasks have no answer at all. For PR, the authors trained four Arabic BERT variants as Sentence-Transformers bi-encoders on the task's training data plus QRCD_v1.1, using MultipleNegativesRankingLoss. They then merged the rankings of the strongest models. For MRC, they fine-tuned nine Arabic pre-trained models on QRCD_v1.2. They added extra training data from the Classical Arabic QUQA set and the MSA ARCD set, combined the best models with score-summing voting, and applied post-processing to the ranked spans.

On the development set, the dense retrievers beat BM25 by a wide margin, but this advantage largely disappeared on the test set. In MRC, an ensemble of AraBERT base and large with post-processing performed best on the development set, yet the single AraBERT base model was slightly better on the test set. The work shows that the dense approach is viable for Classical Arabic, while showing how fragile model selection is when development data is tiny.

Key points

  • PR data: 1,266 topic-segmented Qur'anic passages (QPC) and 199 AyaTEC-derived questions, split 70/10/20. On the development set, BM25 reached 0.17 MAP, while bi-encoder ArabicBERT reached 0.511 and the ArabicBERT + CL-AraBERT ensemble reached 0.534.
  • Re-ranking with the multilingual cross-encoder mmarco-mMiniLMv2-L12-H384-v1 consistently made results worse than the bi-encoders alone.
  • PR test results were far lower: CL-AraBERT scored 0.124 MAP, the ensemble 0.117, and ArabicBERT 0.07. Twenty-one questions scored zero, some of them unanswerable questions that the systems failed to recognise.
  • MRC pAP@10 on the development set: AraBERT base reached 0.402 with QRCD only. AraBERT large reached 0.482 with QUQA (3,382 triplets) added. AraBERT base reached 0.49 with QUQA and ARCD together.
  • MRC ensembles on the development set: all nine models scored 0.466, AraBERT base + large scored 0.517, and the same pair with post-processing scored 0.537. On the test set, AraBERT base scored 0.5 and the ensemble 0.498.

Interesting ideas

  • I1. The PR ensemble first ranks passages that every model retrieved, ordered by their averaged score, and only then fills the remaining slots with the strongest single-model candidates; unanimity acts as a priority tier, not just a weight.
  • I2. Adding in-domain Classical Arabic QA data (QUQA) helped more than half of the models, while general MSA Wikipedia data (ARCD) helped only one; domain match mattered more than volume.
  • I3. Error analysis showed the reader picking spans that merely share words with the question, and missing true answers phrased with synonyms.
  • I4. Ensembles that won on a small development set lost to a single model on the test set, and an off-the-shelf reranker hurt performance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data from witness parsers could be tiered the same way, with arcs all witnesses agree on going first and single-witness arcs used only as lower-confidence filler.
  • I2 (tree): Treebank and silver-data choices for gparse could prefer register-matched text, for example fables and school texts, over larger generic corpora.
  • I3 (answer): Readers should select answers by matching the question's wh-role against frame arguments, and tests should include distractor sentences with high word overlap as negative controls.
  • I4 (learn): Treat gains on small development sets as provisional, and accept a pipeline addition only if it survives a second held-out split and unanswerable-question controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of LeedsLeeds · United Kingdom40 works · 1,822 citations🎓King Abdulaziz UniversityJeddah · Saudi Arabia31 works · 508 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 worksInformation retrieval & search · 473 works

Cite

Cite as Sarah Alnefaie, Abdullah Alsaleh, Eric Atwell, Mohammad Alsalka and Abdulrahman Altahhan. 2023. LKAU23 at Qur'an QA 2023: Using Transformer Models for Retrieving Passages and Finding Answers to Questions from the Qur'an. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.80

@inproceedings{alnefaie2023lkau23,
  title = {{LKAU23 at Qur'an QA 2023: Using Transformer Models for Retrieving Passages and Finding Answers to Questions from the Qur'an}},
  author = {Sarah Alnefaie, Abdullah Alsaleh, Eric Atwell, Mohammad Alsalka and Abdulrahman Altahhan},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.80},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.80}
}

References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Esha Aftab and Muhammad Kamran Malik. 2022. erock at qur’an qa 2022: Contemporary deep neural networks for qur’an based reading comprehension question answers. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 96–103.
  4. Mohammad Mushabbab A Alqahtani. 2019. Quranic Arabic semantic search model based on ontology of concepts. Ph.D. thesis, University of Leeds. find
  5. Kholoud Alsubhi, Amani Jamal, and Areej Alhothali. 2021. Pre-trained transformer-based approach for arabic question answering: A comparative study. arXiv preprint arXiv:2111.05671. ↗
  6. Kholoud Alsubhi, Amani Jamal, and Areej M. Alhothali. 2022. Deep learning-based approach for arabic open domain question answering. PeerJ Computer Science, 8. ↗
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  8. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  9. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. ↗
  10. Rana Malhas and Tamer Elsayed. 2020. Ayatec: Building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 19(6). find
  11. Rana Malhas and Tamer Elsayed. 2022. Arabic machine reading comprehension on the holy qur’an using cl-arabert. Information Processing & Management, 59(6):103068. ↗
  12. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. find
  13. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 79–87, Marseille, France. European Language Resources Association.
  14. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. find
  15. Qur’an QA 2023 Shared Task: Overview of Passage Retrieval and Reading Comprehension Tasks over the Holy Qur’an. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore.
  16. Ali Mostafa and Omar Mohamed. 2022. Gof at qur’an qa 2022: Towards an efficient question answering Dataset for the holy qu’ran in the arabic language using deep learning-based approach. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 104–111. Training 174 Development 25 find
  17. Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj. 2019. Neural arabic question answering. arXiv preprint arXiv:1906.05394. ↗
  18. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. ↗
  19. Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3781–3797, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alnefaie-2023-lkau23-qur-qa-2023-transformer/