Elamur Research⌕
☰
home / papers / 2023 / AHJL at Qur'an QA 2023 Shared Task: Enhancing Passage Retrieval using Sentence Transformer and Translation

AHJL at Qur'an QA 2023 Shared Task: Enhancing Passage Retrieval using Sentence Transformer and Translation

Paper · 2023 · Hessa Alawwad, Lujain Alawwad, Jamilah Alharbi et al.

Summary

The paper addresses Task A (passage retrieval) of the Qur'an QA 2023 Shared Task. Given a Modern Standard Arabic question, a system must rank Qur'anic passages by how likely they are to hold the answer, or report that no answer exists. The authors found that available retrievers handled Arabic poorly, so they translated each question into English with gpt-3.5-turbo. Matching then ran against an English rendering of the verses from the Rowwad Translation Center. Optionally, the system also generated paraphrases of the question.

Retrieval follows a two-stage design. A bi-encoder pulls candidates and a cross-encoder re-ranks them. The authors tested this with Sentence Transformers models and with OpenAI models, all zero-shot. A cutoff on the re-ranker score is used to declare "no answer". Moving from an Arabic sentence encoder to the translated English pipeline raised dev MAP from almost nothing to 0.343. Paraphrasing brought no statistically significant gain. The work shows that a translation pivot can let low-resource or classical-language queries reuse strong English retrieval models without any fine-tuning.

Key points

  • Data:
    • The Qur'anic Passage Collection (QPC) has 1,266 thematic passages.
    • AyaTEC questions are split into 174 train (25 no-answer), 25 dev (4 no-answer) and 52 test.
    • The QRels contain 1,132 gold passage IDs, with "-1" marking unanswerable questions.
  • Model A:
    • Bi-encoder: msmarco-distilbert-base-tas-b.
    • Re-ranker: cross-encoder/ms-marco-MiniLM-L-6-v2.
  • Model B:
    • Embeddings: text-embedding-ada-002.
    • Re-ranker: text-davinci-003.
  • Dev results: The best run was SBERT without paraphrasing, with MAP 0.343 and MRR 0.413.
    • The Arabic encoder medmediani/Arabic-KW-Mdel reached a MAP of only 0.003.
  • Test results: The OpenAI variant reached MAP 0.199 and MRR 0.389 in the results table, while SBERT reached 0.132 and 0.302.
    • The prose claims SBERT had the top test MAP, which contradicts the table.
  • No-answer detection:
    • Paraphrasing was judged non-significant by both a t-test and a Mann-Whitney U test.
    • A re-ranker score threshold of -5 correctly flagged 4 of 7 unanswerable test questions (57%).

Interesting ideas

  • I1. Routing non-English questions through machine translation lets a system reuse mature English retrievers instead of training language-specific ones, and here that change alone accounted for most of the gain.
  • I2. The pipeline combines a cheap bi-encoder for recall with a costlier cross-encoder for precise reordering, a pattern that scales well when the candidate pool is large.
  • I3. Unanswerable questions are handled by a calibrated cutoff on the relevance score rather than by forcing a ranked list.
  • I4. The authors checked whether paraphrase augmentation actually helped, using two significance tests rather than raw score differences, and found no reliable effect.
  • I5. In the error analysis, some questions required inference beyond surface similarity, showing where similarity-only retrieval breaks down.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Translation through the meta-world could use the same pivot logic, mapping foreign input onto Mova's English pipeline and measuring how much meaning survives the round trip.
  • I2 (answer): Story QA readers could shortlist candidate sentences or frames cheaply, then apply full tree- and frame-based reasoning only to the top few, keeping explanations focused.
  • I3 (answer): Readers on tests and fables could abstain explicitly when no frame in the text world matches well enough, instead of guessing.
  • I4 (learn): Evaluations of new augmentation or repair rules should include significance testing, so that small noisy improvements are not mistaken for real progress.
  • I5 (world): This supports Mova's design choice of building an explicit world of entities and events, so that indirect questions can be answered by reasoning rather than by vector proximity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Abdulaziz UniversityJeddah · Saudi Arabia31 works · 508 citations🎓Imam Mohammad ibn Saud Islamic UniversityRiyadh · Saudi Arabia11 works · 51 citations🎓Saudi Electronic UniversityRiyadh · Saudi Arabia4 works · 36 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 worksInformation retrieval & search · 473 works

Cite

Cite as Hessa Alawwad, Lujain Alawwad, Jamilah Alharbi and Abdullah Alharbi. 2023. AHJL at Qur'an QA 2023 Shared Task: Enhancing Passage Retrieval using Sentence Transformer and Translation. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.77

@inproceedings{alawwad2023ahjl,
  title = {{AHJL at Qur'an QA 2023 Shared Task: Enhancing Passage Retrieval using Sentence Transformer and Translation}},
  author = {Hessa Alawwad, Lujain Alawwad, Jamilah Alharbi and Abdullah Alharbi},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.77},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.77}
}

References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2023. Rowwad translation center, the noble qur’an encyclopedia. find
  2. Heba Abdelnasser, Maha Ragab, Reham Mohamed, Alaa Mohamed, Bassant Farouk, Nagwa M El-Makky, and Marwan Torki. 2014. Al-bayan: an arabic question answering system for the holy quran. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 57–64. ↗
  3. Palakorn Achananuparp, Xiaohua Hu, and Xiajiong Shen. 2008. The evaluation of sentence similarity measures. In Data Warehousing and Knowledge Discovery: 10th International Conference, DaWaK 2008 Turin, Italy, September 2-5, 2008 Proceedings 10, pages 305–316. Springer. find
  4. A Adeleke, NA Samsudin, ZA Othman, and SK Ahmad Khalid. 2019. A two-step feature selection method for quranic text classification. Indonesian Journal of Electrical Engineering and Computer Science, 16(2):730–736. ↗
  5. Fatima T AL-Khawaldeh. 2015. Answer extraction for why arabic questions answering systems: Ewaq. World of Computer Science & Information Technology Journal, 5(5). find
  6. Hamza Alami, Abdelkader Mahdaouy, Abdessamad Benlahbib, Noureddine En-Nahnahi, Ismail Berrada, and Said El Alaoui Ouatik. 2023. Daqas: Deep arabic question answering system based on duplicate question detection and machine reading comprehension. Journal of King Saud University-Computer and Information Sciences, page 101709. ↗
  7. Jawharah Saeed N Alasmari. 2020. A Comparative Analysis of The Arabic and English Verb Systems Using the Qur’an Arabic Corpus [A corpus-based study]. Ph.D. thesis, University of Leeds. find
  8. Tahani H Alwaneen, Aqil M Azmi, Hatim A Aboalsamh, Erik Cambria, and Amir Hussain. 2022. Arabic question answering system: a survey. Artificial Intelligence Review, pages 1–47. ↗
  9. Eric Atwell, Claire Brierley, Kais Dukes, Majdi Sawalha, and Abdul-Baquee Sharaf. 2011. An artificial intelligence approach to arabic and islamic content on the internet. In Proceedings of NITS 3rd National Information Technology Symposium, pages 1–8. Leeds. find
  10. Wided Bakari and Mahmoud Neji. 2022. A novel semantic and logical-based approach integrating rte technique in the arabic question–answering. International Journal of Speech Technology, 25(1):1–17. find
  11. Jay M Ponte and W Bruce Croft. 2017. A language modeling approach to information retrieval. In ACM SIGIR Forum, volume 51, pages 202–208. ACM New York, NY, USA. ↗
  12. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics. ↗
  13. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. ↗
  14. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. ↗
  15. Rana Malhas. 2023. Arabic Question Answering on the Holy Qur’an. Doctoral dissertation. ↗
  16. Nils Reimers. 2022. Sbert semantic searchs. find
  17. Doug Semenick. 1990. Tests and measurements: The ttest. Strength & Conditioning Journal, 12(1):36–37. find
  18. M. N. Swar. 2007. Mushaf Al-Tafseel Al-Mawdoo’ee. Dar Al-Fajr Al-Islami. find
  19. Chengxiang Zhai and John Lafferty. 2004. A study of smoothing methods for language models applied to information retrieval. ACM Transactions on Information Systems (TOIS), 22(2):179–214. ↗
  20. Rana Malhas and Tamer Elsayed. 2020. Ayatec: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6):1–21. find
  21. Rana Malhas and Tamer Elsayed. 2022. Arabic machine reading comprehension on the holy qur’an using cl-arabert. Information Processing & Management, 59(6):103068. ↗
  22. Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. Qur’an qa 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 79–87.
  23. Patrick E McKnight and Julius Najab. 2010. Mann-whitney u test. The Corsini encyclopedia of psychology, pages 1–1. ↗
  24. Ensaf Hussein Mohamed and Eyad Mohamed Shokry. 2022. Qsst: A quranic semantic search tool based on word embedding. Journal of King Saud University-Computer and Information Sciences, 34(3):934–945. find
  25. Masnizah Mohd, Faizan Qamar, Idris Al-Sheikh, and Ramzi Salah. 2021. Quranic optical text recognition using deep learning models. IEEE Access, 9:38318– 38330. ↗
  26. Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. 2020. Beyond [CLS] through ranking by generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1722–1727, Online. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alawwad-2023-ahjl-qur-qa-2023-shared/