Elamur Research⌕
☰
home / papers / 2023 / GYM at Qur'an QA 2023 Shared Task: Multi-Task Transfer Learning for Quranic Passage Retrieval and Question Answering with Large Language Models

GYM at Qur'an QA 2023 Shared Task: Multi-Task Transfer Learning for Quranic Passage Retrieval and Question Answering with Large Language Models

Paper · 2023 · Ghazaleh Mahmoudi, Yeganeh Morshedzadeh and Sauleh Eetemadi

Summary

The paper describes the GYM team's entry to the Qur'anQA 2023 shared task, which asks systems to answer Modern Standard Arabic questions over the Quran in two stages. Task A retrieves relevant verse passages. Task B extracts answer spans from a given passage. Labelled data is very scarce, so the authors rely on transfer from larger Arabic resources.

For retrieval, they build an SBERT bi-encoder on AraBERT in two steps. First they adapt it to Quranic text without labels, using TSDAE or SimCSE. Then they train it with supervision on two sources at once: the Arabic part of Mr. TyDi and Quran question-passage pairs, with hard negatives mined by BM25. For span extraction, they fine-tune AraElectra checkpoints already trained on Arabic SQuADv2 or TyDiQA on QRCD v1.2 and also try a score-summing ensemble.

Retrieval stays weak on the test set, and the training scores far exceed the test scores. Reading comprehension works much better: the SQuADv2-based model reaches 46.1% pAP@10 on test, clearly above the baseline. The work shows how much carefully chosen transfer data helps in a low-resource domain.

Key points

  • Task A data: 174 training, 25 development and 52 test questions, with 972 and 160 question-passage pairs in train and dev. Thirty questions have no answer in the Quran.

  • Task A training:

    • Mr. TyDi (Arabic) triplets are trained with multiple negatives ranking loss.
    • Quranic pairs use either contrastive loss, with negatives from the top 1000 BM25 hits, or triplet loss, with negatives from the top 100 BM25 hits.
    • Training ran for 3 epochs with batch size 64, taking about 48 minutes on an RTX 3090.

  • Task A results:

    • Best test result is AraBERT-TSDAE-Contrastive, with MAP@10 0.0545 and MRR@10 0.1581.
    • On dev, AraBERT-SimCSE-Contrastive has the best MAP@10 (0.1459).
    • On train, AraBERT-SimCSE-Contrastive reaches MAP@10 0.6522, a large overfitting gap.
    • Contrastive loss generally beat triplet loss.

  • Task B data: QRCD v1.2, in which about 15% of questions have no answer.

  • Task B results (pAP@10, dev / test):

    ModelDevTest
    AraElectra-SQuADv20.4850.461
    Ensemble0.4810.458
    AraElectra-TyDiQA0.4310.430
    Baseline0.2550.326

    The authors describe the gain over the baseline as 23%.

Interesting ideas

  • I1. Hard negatives for contrastive training are taken from a cheap lexical retriever (BM25): passages that look similar on the surface but are not gold answers. This teaches the model to separate real relevance from mere word overlap.
  • I2. In multi-task training, each dataset keeps the loss that fits its format: ranking loss for triplets, and contrastive or triplet loss for labelled pairs. This lets heterogeneous resources be mixed without forcing them into one schema.
  • I3. The ensemble sums the scores of spans that two fine-tuned models both propose, while keeping unshared spans at their original score. Agreement therefore boosts a candidate without discarding minority answers.
  • I4. Before supervised training, the encoder is adapted on raw target-domain text with a denoising objective (TSDAE). The authors credit this step with better ranking of relevant items, reflected in higher MRR.
  • I5. The error analysis splits cases by type. Retrieval succeeds when the question and passage share named entities and fails when only one obscure passage is relevant. Unanswerable questions break an untuned score threshold.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Near-miss distractors could be mined with a lexical index and used as negatives when training frame or answer rankers, so the model cannot win on shared words alone.
  • I2 (learn): When silver data, gold treebanks and LLM-teacher labels are combined, each source could get its own objective rather than being flattened into one label type.
  • I3 (answer): Candidate answers from several readers could be merged the same way, so agreement raises confidence while single-witness answers stay available and traceable.
  • I4 (tree): The gparse transformer encoder could first be adapted on unlabelled fable and textbook text before treebank fine-tuning, to help with archaic or literary style.
  • I5 (answer): QA evaluation on tales could report entity-overlap and no-answer subsets separately, and use questions with no answer as explicit negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓Iran University of Science and TechnologyTehran · Iran4 works · 65 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 worksInformation retrieval & search · 473 works

Cite

Cite as Ghazaleh Mahmoudi, Yeganeh Morshedzadeh and Sauleh Eetemadi. 2023. GYM at Qur'an QA 2023 Shared Task: Multi-Task Transfer Learning for Quranic Passage Retrieval and Question Answering with Large Language Models. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.79

@inproceedings{mahmoudi2023gym,
  title = {{GYM at Qur'an QA 2023 Shared Task: Multi-Task Transfer Learning for Quranic Passage Retrieval and Question Answering with Large Language Models}},
  author = {Ghazaleh Mahmoudi, Yeganeh Morshedzadeh and Sauleh Eetemadi},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.79},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.79}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Basem Ahmed, Motaz Saad, and Eshrag A. Refaee. 2022. QQATeam at qur’an QA 2022: Fine-tunning Arabic QA models for qur’an QA task. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 130–135, Marseille, France. European Language Resources Association.
  2. Zeyad Ahmed. 2023a. Arabic Machine Reading Comprehension: Effective Models and Introducing Arabic-SQuAD v2.0. https://github.com/zeyadahmed10/Arabic-MRC, https://huggingface.co/ZeyadAhmed/ Original-AraElectra-Arabic-SQuADv2-QA. date: 2021-11-04T18:03:17Z. ↗
  3. Rana Malhas and Tamer Elsayed. 2020. Ayatec: Building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 19(6). find
  4. Rana Malhas and Tamer Elsayed. 2022. Arabic machine reading comprehension on the holy qur’an using cl-arabert. Information Processing Management, 59(6):103068. ↗
  5. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. find
  6. Qur’an QA 2022: Overview of the First Shared Task on Question Answering over the Holy Qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022), pages 79–87.
  7. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2023. find
  8. Qur’an QA 2023 Shared Task: Overview of Passage Retrieval and Reading Comprehension Tasks over the Holy Qur’an. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore.
  9. Zeyad Ahmed. 2023b. Arabic SQuAD v2.0 Dataset based on the popular SQuADv2.0 with unanswered questions for more challenging task. https://huggingface.co/datasets/ ZeyadAhmed/Arabic-SQuADv2.0. Original-date: 2022-06-29T18:03:17Z. ↗
  10. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  11. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  12. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics. ↗
  13. Xingping Dong and Jianbing Shen. 2018. Triplet loss in siamese network for object tracking. In Proceedings of the European Conference on Computer Vision (ECCV). find
  14. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP).
  15. Rana R Malhas. 2023. ARABIC QUESTION ANSWER-ING ON THE HOLY QUR’AN. Ph.D. thesis. find
  16. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  17. Marwan N. Swar. 2007. Mushaf Al-Tafseel Al-Mawdoo’ee. Dar Al-Fajr Al-Islami, Damascus. find
  18. Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. TSDAE: Using transformer-based sequential denoising auto-encoderfor unsupervised sentence embedding learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 671–688, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  19. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 127– 137, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  20. R. Hadsell, S. Chopra, and Y. LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. find
  21. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. ArXiv, abs/1705.00652. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mahmoudi-2023-gym-qur-qa-2023-shared/