Elamur Research⌕
☰
home / papers / 2022 / DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain

DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain

Paper · 2022 · Damith Premasiri, Tharindu Ranasinghe, Wajdi Zaghouani et al.

Summary

The paper describes the DTW team's entry to the Qur'an QA 2022 shared task. The task is extractive (span-prediction) reading comprehension over Qur'anic passages, a domain with very little annotated data. The organisers supplied QRCD, which has 1,093 question–passage pairs and 1,337 question–passage–answer triplets. Its training split holds only 710 pairs, too few to fine-tune large neural readers reliably.

The authors compared seven Arabic-capable pre-trained transformers, using the standard start/end span head. They tested two remedies for the data shortage:

  • Self ensemble learning, which turned out to give steadier rather than higher scores.
  • Transfer learning: first fine-tuning on the much larger Arabic MRC collection SOQAL, then on QRCD.

Transfer learning lifted AraELECTRA-discriminator from 0.528 to 0.616 pRR on the dev set, though other models did not gain. This model was also the best submitted system on the test set, at roughly 0.49–0.495 pRR.

The work matters because it shows that borrowing a related, larger resource in the same language can make modern readers usable in a niche, low-resource domain. Code, a pip package, HuggingFace models and a Docker image are released.

Key points

  • Data: QRCD was split 65/10/25% into 710/109/274 question–passage pairs (861/128/348 triplets). The transfer source was SOQAL, which combines:
    • ARCD: 1,395 crowdsourced questions on Wikipedia.
    • A machine translation of SQuAD: 48,344 questions.
  • Models compared: camelbert-mix, camelbert-ca, mbert-cased, mbert-uncased, AraELECTRA-generator, AraELECTRA-discriminator and AraBERTv2.
  • Training setup: batch size 8, Adam with learning rate 2e−5, 10% linear warm-up and 5 epochs.
  • Dev results without transfer: AraELECTRA-discriminator (0.528 pRR) and camelbert-mix (0.520) were strongest, and AraELECTRA-generator was weakest (0.364).
  • Effect of transfer learning: it gave AraELECTRA-discriminator 0.616 pRR and 0.609 F1@1 on dev, but did not help the other models tested.
  • Test results: AraELECTRA-discriminator with transfer and ensembling scored 0.495 pRR. Camelbert-mix without transfer fell sharply to 0.290 pRR, while the same model with transfer reached 0.408.

Interesting ideas

  • I1. Two-stage fine-tuning (a large related span-QA set first, then the tiny target set) rescued a reader that the small domain alone could not train well.
  • I2. The authors found that ensembling identical-architecture models mostly reduced run-to-run variance rather than raising scores, and reported it honestly in those terms.
  • I3. A machine-translated version of a big benchmark served as useful auxiliary training data despite translation noise.
  • I4. Gains from transfer were model-specific, and a model that looked fine on dev collapsed on test without transfer, so dev scores on tiny sets can mislead.
  • I5. Scoring with a ranked metric that gives partial credit (pRR) alongside exact match captures answers that are close but not identical.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Readers for fables and tales could first be tuned on broader reading-comprehension material and only then specialised on our small tale QA sets.
  • I2 (learn): When training parsers or readers, report the spread across several runs and treat stability as its own metric, separate from accuracy.
  • I3 (learn): Noisy, automatically produced resources can still be useful as silver pre-training data, provided final tuning and evaluation rely on clean gold data.
  • I4 (learn): Keep a held-out set and negative controls for every small-data component, so that overfitting to a few hundred examples is caught before release.
  • I5 (answer): Grading school-test and tale QA answers with ranked, overlap-based credit would show near-misses that exact match hides.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar144 works · 14,175 citations🎓University of WolverhamptonWolverhampton · United Kingdom36 works · 6,331 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Question answering · 487 worksLow-resource languages · 287 works

Cite

Cite as Damith Premasiri, Tharindu Ranasinghe, Wajdi Zaghouani and Ruslan Mitkov. 2022. DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{premasiri2022dtw,
  title = {{DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain}},
  author = {Damith Premasiri, Tharindu Ranasinghe, Wajdi Zaghouani and Ruslan Mitkov},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.10/}
}

References

42 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadani, N., Mozafari, J., Fatemi, A., Nematbakhsh, M. A., and Kazemi, A. (2021). Parsquad: Machine translated squad dataset for persian question answering. In 2021 7th International Conference on Web Research (ICWR), pages 163–168. find
  2. Antoun, W., Baly, F., and Hajj, H. (2020). AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France, May. European Language Resource Association.
  3. Antoun, W., Baly, F., and Hajj, H. (2021). Ara-ELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics. find
  4. Baradaran, R., Ghiasi, R., and Amirkhani, H. (2022). A survey on machine reading comprehension systems. Natural Language Engineering, page 1–50. find
  5. Charniak, E., Altun, Y., Braz, R. d. S., Garrett, B., Kosmala, M., Moscovich, T., Pang, L., Pyo, C., Sun, Y., Wy, W., Yang, Z., Zeller, S., and Zorn, L. (2000). Reading comprehension programs in a statistical-language-processing class. In Proceedings of the 2000 ANLP/NAACL Workshop on Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Sytems - Volume 6, ANLP/NAACL-ReadingComp ’00, page 1–5, USA. Association for Computational Linguistics. find
  6. Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., and Gardner, M. (2021). A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599–4610, Online, June. Association for Computational Linguistics.
  7. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171– 4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. d’Hoffschmidt, M., Belblidia, W., Heinrich, Q., Brendlé, T., and Vidal, M. (2020). FQuAD: French question answering dataset. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1193–1208, Online, November. Association for Computational Linguistics.
  8. Efimov, P., Chertok, A., Boytsov, L., and Braslavski, P. (2020). Sberquad – russian reading comprehension dataset: Description and analysis. In Avi Arampatzis, et al., editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 3– 15, Cham. Springer International Publishing. find
  9. Giorgi, S., Zavarella, V., Tanev, H., Stefanovitch, N., Hwang, S., Hettiarachchi, H., Ranasinghe, T., Kalyan, V., Tan, P., Tan, S., Andrews, M., Hu, T., Stoehr, N., Re, F. I., Vegh, D., Atzenhofer, D., Curtis, B., and Hürriyetoğlu, A. (2021). Discovering black lives matter events in the United States: Shared task 3, CASE 2021. In Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021), pages 218–227, Online, August. Association for Computational Linguistics. find
  10. Hermann, K. M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. (2015). Teaching machines to read and comprehend. In C. Cortes, et al., editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
  11. Hettiarachchi, H. and Ranasinghe, T. (2020). InfoMiner at WNUT-2020 task 2: Transformer-based covid-19 informative tweet extraction. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pages 359–365, Online, November. Association for Computational Linguistics. find
  12. Hirschman, L., Light, M., Breck, E., and Burger, J. D. (1999). Deep read: A reading comprehension system. ACL ’99, page 325–332, USA. Association for Computational Linguistics. find
  13. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., and Habash, N. (2021). The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92–104, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics.
  14. Jayakody, J. A. T. K., Gamlath, T. S. K., Lasantha, W. A. N., Premachandra, K. M. K. P., Nugaliyadde, A., and Mallawarachchi, Y. (2016). “mahoshadha”, the sinhala tagged corpus based question answering system. In Suresh Chandra Satapathy et al., editors, Proceedings of First International Conference on Information and Communication Technology for Intelligent Systems: Volume 1, pages 313–322, Cham. Springer International Publishing. find
  15. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020). Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
  16. Lee, J. Y., Dernoncourt, F., and Szolovits, P. (2018). Transfer learning for named-entity recognition with neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). find
  17. Lehnert, W. G. (1977). The process of question answering. Yale University. find
  18. Liu, S., Zhang, X., Zhang, S., Wang, H., and Zhang, W. (2019). Neural machine reading comprehension: Methods and trends. Applied Sciences, 9(18). find
  19. Malhas, R. and Elsayed, T. (2020). Ayatec: Building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 19(6), oct. find
  20. Malhas, R., Mansour, W., and Elsayed, T. (2022). find
  21. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  22. Mozannar, H., Maamary, E., El Hajal, K., and Hajj, H. (2019). Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy, August. Association for Computational Linguistics.
  23. Nguyen, T. Q. and Chiang, D. (2017). Transfer learning across low-resource, related languages for neural machine translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 296– 301, Taipei, Taiwan, November. Asian Federation of Natural Language Processing.
  24. Plum, A., Ranasinghe, T., Jones, S., Orasan, C., and Mitkov, R. (2022). Biographical: A semi-supervised relation extraction dataset. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain. Association for Computing Machinery. find
  25. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November. Association for Computational Linguistics.
  26. Ranasinghe, T. and Hettiarachchi, H. (2020). BRUMS at SemEval-2020 task 12: Transformer based multilingual offensive language identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1906–1915, Barcelona (online), December. International Committee for Computational Linguistics. find
  27. Ranasinghe, T. and Zampieri, M. (2020). Multilingual offensive language identification with cross-lingual embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5838–5844, Online, November. Association for Computational Linguistics.
  28. Ranasinghe, T., Orasan, C., and Mitkov, R. (2021). An exploratory analysis of multilingual word-level quality estimation with cross-lingual transformers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 434–440, Online, August. Association for Computational Linguistics. find
  29. Richardson, M., Burges, C. J., and Renshaw, E. (2013). MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203, Seattle, Washington, USA, October. Association for Computational Linguistics.
  30. Riloff, E. and Thelen, M. (2000). A rule-based question answering system for reading comprehension tests. In Proceedings of the 2000 ANLP/-NAACL Workshop on Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Sytems - Volume 6, ANLP/NAACL-ReadingComp ’00, page 13–19, USA. Association for Computational Linguistics. find
  31. Rouws, N. J., Vakulenko, S., and Katrenko, S. (2022). Dutch squad and ensemble learning for question answering from labour agreements. In Luis A. Leiva, et al., editors, Artificial Intelligence and Machine Learning, pages 155–169, Cham. Springer International Publishing. find
  32. Sagi, O. and Rokach, L. (2018). Ensemble learning: A survey. WIREs Data Mining and Knowledge Discovery, 8(4):e1249. find
  33. Suzuki, J., Sasaki, Y., and Maeda, E. (2002). SVM answer selection for open-domain question answering. In COLING 2002: The 19th International Conference on Computational Linguistics. find
  34. Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., and Suleman, K. (2017). NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200, Vancouver, Canada, August. Association for Computational Linguistics.
  35. Uyangodage, L., Ranasinghe, T., and Hettiarachchi, H. (2021). Transformers to fight the COVID-19 infodemic. In Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 130–135, Online, June. Association for Computational Linguistics.
  36. Welbl, J., Liu, N. F., and Gardner, M. (2017). Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark, September. Association for Computational Linguistics.
  37. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October. Association for Computational Linguistics.
  38. Yamada, I., Asai, A., Shindo, H., Takeda, H., and Matsumoto, Y. (2020). LUKE: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442–6454, Online, November. Association for Computational Linguistics. find
  39. Yen, S.-J., Wu, Y.-C., Yang, J.-C., Lee, Y.-S., Lee, C.-J., and Liu, J.-J. (2013). A support vector machine-based context-ranking model for question answering. Information Sciences, 224:77–87. find
  40. Zeng, C., Li, S., Li, Q., Hu, J., and Hu, J. (2020). A survey on machine reading comprehension—tasks, evaluation metrics and benchmark datasets. Applied Sciences, 10(21). find
  41. Zhang, Z., Wu, Y., Zhou, J., Duan, S., Zhao, H., and Wang, R. (2020). Sg-net: Syntax-guided machine reading comprehension. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9636– 9643, Apr. find
  42. Zhang, Z., Yang, J., and Zhao, H. (2021). Retrospective reader for machine reading comprehension. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14506–14514, May. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/premasiri-2022-dtw-qur-qa-2022-utilising/