Elamur Research⌕
☰
home / papers / 2022 / LK2022 at Qur'an QA 2022: Simple Transformers Model for Finding Answers to Questions from Qur'an

LK2022 at Qur'an QA 2022: Simple Transformers Model for Finding Answers to Questions from Qur'an

Paper · 2022 · Abdullah Alsaleh, Saud Althabiti, Ibtisam Alshammari et al.

Summary

This paper describes a team entry to the Qur'an QA 2022 shared task. Given a Qur'anic passage and a question in Modern Standard Arabic, the system must extract answer spans from the passage. The authors used the Simple Transformers question-answering library and compared three BERT-style Arabic encoders: AraBERT, CAMeL-BERT and ArabicBERT. The training data was the QRCD dataset, converted from its JSON Lines layout into the context/qas structure the library expects. The text was not preprocessed. For each question, the system returns five candidate spans ranked by probability, and these are scored mainly with partial Reciprocal Rank (pRR). The only settings the authors tuned were the learning rate, the batch size and the epoch count. On the development set, AraBERT V0.2 (large) performed best, so it was submitted for the test run. It reached moderate test scores, with a noticeable drop from development. The error analysis is the most useful part. The model succeeds when the question and passage share words or synonyms and struggles when they do not. Some "wrong" outputs look like valid paraphrases that the gold answers miss.

Key points

  • QRCD contains 1093 question–passage pairs: 710 for training, 109 for development and 274 unanswered pairs for testing. Of these questions, 99 in training and 15 in development have more than one gold answer.
  • The encoders compared were:
    • AraBERT V0.2 large, the variant without Farasa segmentation, which the authors had chosen in earlier Quran semantic-similarity work.
    • CAMeL-BERT, using its Classical Arabic base variant pretrained on OpenITI.
    • ArabicBERT large, pretrained on OSCAR and Wikipedia. The mega size could not be run on their hardware.
  • AraBERT V0.2 scored best on development, with pRR 0.59, EM 0.34 and F1 0.55. On the test set it scored pRR 0.45, EM 0.16 and F1 0.42.
  • The authors trained for up to 25 epochs and found that roughly 5–7 epochs were enough. All experiments ran on Google Colab with CUDA.
  • In the related-work survey, earlier Quranic QA systems mostly returned whole verses or handled only narrow question types. These systems included Al-Bayan, an ALICE chatbot and an ontology plus word2Vec search.

Interesting ideas

  • I1. The system outputs a ranked list of five scored spans instead of a single answer, and pRR gives partial credit for overlap and rank, so near-misses still count.
  • I2. The error analysis splits cases by whether the question and passage share words or synonyms, and the model fails mostly when they share none.
  • I3. When the authors inspected "incorrect" predictions, they found paraphrases that were arguably correct, which exposed gaps in the gold annotation.
  • I4. The same question can have different correct answers depending on the passage, so the answer must be grounded in the given text, not in general knowledge.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Readers on fables and school tests could return several candidate answers, each with a score and a justification path. They could then be evaluated with a pRR-style metric that separates "close but ranked second" from "totally wrong."
  • I2 (learn): A held-out slice of questions with no lexical overlap can act as a negative control. This would check that the reader truly reasons over frames rather than matching surface words.
  • I3 (learn): Disagreements with gold should be triaged before being counted as errors. Plausible paraphrase answers can be flagged as candidate gold additions, keeping the silver/gold data honest.
  • I4 (world): Answers should be resolved against the world built from the current text, with compiled background knowledge used only as support. This prevents a memorized fact from overriding what the passage actually says.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksQuestion answering · 487 works

Cite

Cite as Abdullah Alsaleh, Saud Althabiti, Ibtisam Alshammari, Sarah Alnefaie, Sanaa Alowaidi, Alaa Alsaqer, Eric Atwell, Abdulrahman Altahhan and Mohammad Alsalka. 2022. LK2022 at Qur'an QA 2022: Simple Transformers Model for Finding Answers to Questions from Qur'an. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alsaleh2022lk2022,
  title = {{LK2022 at Qur'an QA 2022: Simple Transformers Model for Finding Answers to Questions from Qur'an}},
  author = {Abdullah Alsaleh, Saud Althabiti, Ibtisam Alshammari, Sarah Alnefaie, Sanaa Alowaidi, Alaa Alsaqer, Eric Atwell, Abdulrahman Altahhan and Mohammad Alsalka},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.14/}
}

References

22 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelnasser, H., Ragab, M., Mohamed, R., Mohamed, A., Farouk, B., El-Makky, N. M., and Torki, M. (2014). Al-bayan: an arabic question answering system for the holy quran. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 57–64.
  2. AbuShawar, B. and Atwell, E. (2016). Usefulness, localizability, humanness, and language-benefit: additional evaluation criteria for natural language dialogue systems. International Journal of Speech Technology, 19(2):373–383. find
  3. Al-Tabari, M. B. J. (1954).   à @Q ®Ë@ ø @ ÉK ðAK á« àAJ J.Ë@ ©ÓAg. . Dar Al-Fikr. find
  4. Alqahtani, M. M. A. (2019). Quranic Arabic Semantic Search Model Based on Ontology of Concepts. Ph.D. thesis, University of Leeds. find
  5. Alsaleh, A., Atwell, E., and Altahhan, A. (2021). Quranic verses semantic relatedness using AraBERT. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 185–190, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics.
  6. Alsubhi, K., Jamal, A., and Alhothali, A. (2021). Pre-trained transformer-based approach for arabic question answering: A comparative study. arXiv preprint arXiv:2111.05671. ↗
  7. Altammami, S., Atwell, E., and Alsalka, A. (2020). Towards a joint ontology of quran and hadith. International Journal on Islamic Applications in Computer Science And Technology. find
  8. Antoun, W., Baly, F., and Hajj, H. (2020). AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France, May. European Language Resource Association. ↗
  9. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  10. El-Khair, I. A. (2016). 1.5 billion words arabic corpus. ArXiv, abs/1611.04033. ↗
  11. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., and Habash, N. (2021). The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online), April. Association for Computational Linguistics.
  12. Mahdi, A. F. (2021). Survey: using bert model for arabic question answering system. Turkish Journal of Computer and Mathematics Education (TURCO-MAT), 12(13):723–729. find
  13. Malhas, R. and Elsayed, T. (2020). Ayatec: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6):1–21. find
  14. Malhas, R., Mansour, W., and Elsayed, T. (2022). find
  15. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  16. Mozannar, H., Hajal, K. E., Maamary, E., and Hajj, H. (2019). Neural arabic question answering. arXiv preprint arXiv:1906.05394. ↗
  17. Nigst, L., Romanov, M., Savant, S. B., Seydi, M., and Verkinderen, P. (2020). Openiti: a machine-readable corpus of islamicate texts, Jun. find
  18. Ortiz Suárez, P. J., Romary, L., and Sagot, B. (2020). A monolingual approach to contextualized word embeddings for mid-resource languages. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
  19. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. ↗
  20. Safaya, A., Abdullatif, M., and Yuret, D. (2020). KUI-SAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online), December. International Committee for Computational Linguistics. find
  21. Yang, Y., Yih, W.-t., and Meek, C. (2015). Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2013–2018.
  22. Zeroual, I., Goldhahn, D., Eckart, T., and Lakhouaja, A. (2019). Osian: Open source international arabic news corpus - preparation and integration into the clarin-infrastructure. In WANLP@ACL 2019. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alsaleh-2022-lk2022-qur-qa-2022-simple/