Elamur Research⌕
☰
home / papers / 2022 / QQATeam at Qur'an QA 2022: Fine-Tunning Arabic QA Models for Qur'an QA Task

QQATeam at Qur'an QA 2022: Fine-Tunning Arabic QA Models for Qur'an QA Task

Paper · 2022 · Basem Ahmed, Motaz Saad and Eshrag A. Refaee

Summary

This paper reports one team's entry in the Qur'an QA 2022 shared task. The task is extractive question answering: given a question in Modern Standard Arabic and a passage of Qur'anic verses in Classical Arabic, the system must return answer spans. Instead of building a QA model from scratch, the authors took two Arabic extractive QA checkpoints from Hugging Face, both AraELECTRA-base models already trained on reading-comprehension data (ARCD and TyDi QA). They fine-tuned these on the shared-task data and also tried a hybrid of the two. A second set of training runs combined the train and dev splits with extra examples made by hand, where only the question was rewritten. The TyDi-based model performed best, reaching pRR 0.63 and F1 0.59 on dev and pRR 0.56 and F1 0.51 on test. Exact match stayed low, at most 0.34, which shows how hard it is to recover precise spans in this domain. The work is useful mainly as a baseline on a common benchmark, since earlier Qur'anic QA systems had never been compared on a shared test set.

Key points

  • The dataset derives from AyaTEC and contains 1,093 question–passage pairs (1,337 question–passage–answer triplets). It is split 710/109/274 pairs (65/10/25%) across train, dev and test.

  • Base models: AraElectra-ARCD, trained on ARCD (1,395 crowd-written questions on Wikipedia), and AraElectra-Artydiqa, trained on TyDi QA. A hybrid of the two was also submitted.

  • Augmentation was done manually and touched only the questions. Paraphrases were produced by changing word order, using synonyms for the object asked about, adding function words, or switching to a different question word.

  • On test, after training on train+dev+augmented data, the three systems scored as follows:

    ModelpRREMF1@1
    AraElectra-Artydiqa≈0.559≈0.244≈0.513
    Hybrid≈0.535—≈0.500
    AraElectra-ARCD≈0.526—≈0.462

  • For future work the authors propose two additions. One is a retrieval stage that ranks passages with the question used as a query. The other is using Qur'anic commentaries to pick the verse most likely to hold the answer.

Interesting ideas

  • I1. Augmenting only the question side, with the answer span left unchanged, gives extra training pairs whose gold labels stay correct and teaches the model that different wordings ask the same thing.
  • I2. The authors chose between starting checkpoints by what each had been trained on earlier, and the one trained on the broader multilingual TyDi QA transferred better than the narrower ARCD one.
  • I3. The paper reports a partial-credit ranking metric (pRR) next to strict exact match. The large gap between them shows the system usually finds roughly the right region but rarely the exact span boundaries.
  • I4. The proposed retrieve-then-read pipeline first ranks candidate passages against the question and then runs span extraction only on the top passage.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Our fable and school-test QA could have each question rewritten with reordered words, synonyms and alternative wh-words. We would then check that the frame-based answer stays the same, which turns paraphrase into a robustness test.
  • I2 (learn): When choosing a teacher or seed model for silver data, compare candidates by how diverse their earlier training data was. Pick by measured transfer on a small in-domain dev set, not by assumption.
  • I3 (answer): Our QA evaluation could score both partial overlap and exact match of answer spans. A persistent gap between the two would indicate span-boundary errors in the answer extraction from the tree rather than misunderstanding of the text.
  • I4 (world): For long texts, the world model could first pick which events or entities are relevant to the question before detailed answering. This keeps every reasoning step small and easy to explain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Basem Ahmed, Motaz Saad and Eshrag A. Refaee. 2022. QQATeam at Qur'an QA 2022: Fine-Tunning Arabic QA Models for Qur'an QA Task. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ahmed2022qqateam,
  title = {{QQATeam at Qur'an QA 2022: Fine-Tunning Arabic QA Models for Qur'an QA Task}},
  author = {Basem Ahmed, Motaz Saad and Eshrag A. Refaee},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.16/}
}

References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelnasser, H., Ragab, M., Mohamed, R., Mohamed, A., Farouk, B., El-Makky, N. M., and Torki, M. (2014). Al-bayan: an arabic question answering system for the holy quran. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 57–64.
  2. Alsubhi, K., Jamal, A., and Alhothali, A. (2021). Pre-trained transformer-based approach for Arabic question answering: A comparative study. arXiv preprint arXiv:2111.05671. ↗
  3. Alwaneen, T. H., Azmi, A. M., Aboalsamh, H. A., Cambria, E., and Hussain, A. (2021). Arabic question answering system: a survey. Artificial Intelligence Review, pages 1–47. ↗
  4. Antoun, W., Baly, F., and Hajj, H. (2020). Araelectra: Pre-training text discriminators for arabic language understanding. ↗
  5. Bouziane, A., Bouchiha, D., Doumi, N., and Malki, M. (2015). Question answering systems: Survey and trends. Procedia Computer Science, 73:366– 375. International Conference on Advanced Wireless Information and Communication Technologies (AWICT 2015). find
  6. Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., and Palomaki, J. (2020). TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics. ↗
  7. Hamdelsayed, M. A. and Atwell, E. (2016). Islamic applications of automatic question-answering. find
  8. Hamed, S. K. and Ab Aziz, M. J. (2016). A question answering system on holy quran translation based on question expansion technique and neural network classification. J. Comput. Sci., 12(3):169–177. ↗
  9. Hammo, B., Abu-Salem, H., Lytinen, S. L., and Evens, M. (2002). QARAB: A: Question answering system to support the arabic language. In Proceedings of the ACL-02 workshop on Computational approaches to semitic languages.
  10. Magdy, A. and Shaheen, M. (2012). A survey of arabic question answering: Challenges, tasks, approaches, tools, and future trends. 10. find
  11. Malhas, R. and Elsayed, T. (2020). AyaTEC: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6):1–21. find
  12. Malhas, R. and Elsayed, T. (2022). official repository of Qur’an QA Shared Task. https://gitlab.com/bigirqu/quranqa, February. ↗
  13. Malhas, R., Mansour, W., and Elsayed, T. (2022). find
  14. Qurán QA 2022: Overview of the first shared task on question answering over the holy qurán. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  15. Meters, C. (2022). Religion of the world. https://countrymeters.info/en/World#religion, journal=Religion of the World, Apr. ↗
  16. Mozannar, H., Maamary, E., El Hajal, K., and Hajj, H. (2019a). Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy, August. Association for Computational Linguistics.
  17. Mozannar, H., Maamary, E., El Hajal, K., and Hajj, H. (2019b). Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy, August. Association for Computational Linguistics.
  18. Prager, J. (2014). Question answering. https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199573691.001.0001/oxfordhb-9780199573691-e-003, journal=Oxford Handbooks Online, Apr. ↗
  19. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November. Association for Computational Linguistics. ↗
  20. Refaee, E. A. A. (2016). Sentiment analysis for microblogging platforms in Arabic. Ph.D. thesis, Heriot-Watt University. find
  21. Su, D., Xu, Y., Winata, G. I., Xu, P., Kim, H., Liu, Z., and Fung, P. (2019). Generalizing question answering system with pre-trained language model fine-tuning. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 203– 211, Hong Kong, China, November. Association for Computational Linguistics. find
  22. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ahmed-2022-qqateam-qur-qa-2022-fine/