Elamur Research⌕
☰
home / papers / 2022 / Harnessing Multilingual Resources to Question Answering in Arabic

Harnessing Multilingual Resources to Question Answering in Arabic

Paper · 2022 · Khalid Alnajjar and Mika Hamalainen

Summary

The paper addresses extractive question answering over the Qur'an for the Qur'an QA 2022 shared task. Given a question and a Qur'anic passage, the system must locate the start and end of the answer span. The public part of the QRCD dataset is small, and the authors work around this in three ways:

  • Base model. They start from multilingual BERT, partly because it was trained on Modern Standard Arabic Wikipedia and not on dialect-heavy web text.
  • Domain adaptation. They continue pretraining it on a crawled religious corpus of Tafseer commentary and Fatwa rulings, plus the full Qur'an.
  • Fine-tuning data. They fine-tune on QRCD together with general QA sets in other languages.

On top of the span predictor they add two steps:

  • Post-processing. Rules stretch answers that are too short for their question type and merge overlapping candidates.
  • Similarity fallback. An AraBERT-based step adds the verse most similar to the question as an extra candidate.

The domain-adapted model outperformed Arabic-specific BERTs when all were trained on QRCD alone. Post-processing gave the largest gain on the development set. Test scores were far lower than development scores. The work shows that foreign-language task data plus in-domain pretraining can partly make up for a tiny target dataset.

Key points

  • QRCD has 1,337 question–passage–answer triplets, split 65/10/25 into train, dev and test. The public training and development portions contain only 710 and 109 samples.
  • The domain corpus comes from six crawled sites, for example quran-tafseer.com (49,888 pages) and islamweb.net (30,691 pages).
    • Diacritics and punctuation were stripped with UralicNLP.
    • Pretraining used MLM and NSP for 3 epochs.
  • Fine-tuning added SQuAD, MLQA and XQuAD. MLQA and XQuAD include Arabic.
  • With QRCD-only training, the adapted model reached pRR 0.347. This beat KUISAIL large BERT (0.330), KUISAIL base BERT (0.286) and CAMeLBERT Quarter (0.274).
  • On dev, post-processing lifted pRR from 0.540 to 0.700. Note that this model had dev data in its training, so these dev figures are optimistic.
  • The best test run reached pRR 0.409 and F1@1 0.364.

Interesting ideas

  • I1. Gather per-question-type statistics on answer length (min, average, max) from training data, grouped by interrogative word, and widen predicted spans that fall short of the average up to the nearest sentence boundary. This is a cheap structural prior that fixes truncated answers.
  • I2. When several predicted spans overlap, merge them into one span from the earliest start to the latest end, rather than reporting near-duplicates.
  • I3. Keep a fallback candidate (the sentence most similar to the question), added only when the main system returns few answers and has not already proposed it.
  • I4. Choose the pretraining source by how close its language variety is to the target text, treating mixed dialect data as noise rather than as extra volume.
  • I5. Format in-domain text for continued pretraining so it mirrors the downstream task, such as question followed by answer, or passage before its explanation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-test and fable QA readers could keep length/shape expectations per wh-type (who → one entity, why → a clause) and flag or extend answers that violate them, with the rule visible in the explanation.
  • I2 (answer): When multiple frames or tree nodes support overlapping answer spans, the reader can merge them into a single answer and record which sources were combined.
  • I3 (answer): A clearly labeled low-confidence fallback could be the sentence whose frames share the most roles with the question frame, used only when the frame-matching reader finds too few answers.
  • I4 (learn): When picking text for silver data or teacher prompts, prefer sources whose register matches the target (e.g., narrative prose for tales) over bigger but mixed corpora.
  • I5 (tree): Domain-adapting gparse on tale and textbook text arranged like the QA inputs it will later parse may help it handle question syntax and narrative passages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Information Systems · 523 works

🧭 Topics 3 Multilingual & cross-lingual NLP · 467 works Question answering · 242 works Semitic languages · 1,484 works

Cite

Cite as Khalid Alnajjar and Mika Hamalainen. 2022. Harnessing Multilingual Resources to Question Answering in Arabic. arXiv:2205.08024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2205.08024 · arXiv 2205.08024

@misc{alnajjar2022harnessing,
  title = {{Harnessing Multilingual Resources to Question Answering in Arabic}},
  author = {Khalid Alnajjar and Mika Hamalainen},
  year = {2022},
  doi = {10.48550/arxiv.2205.08024},
  eprint = {2205.08024},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.08024}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadji, J., Ortiz Suarez, P., Romary, L., and Sagot, B. (2022). Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. arXiv e-prints, page arXiv:2201.06642, January. ↗ our card
  2. Abdelali, A., Darwish, K., Durrani, N., and Mubarak, H. (2016). Farasa: A fast and furious segmenter for arabic. In 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 11–16. Association for Computational Linguistics. our card
  3. Abdelnasser, H., Ragab, M., Mohamed, R., Mohamed, A., Farouk, B., El-Makky, N., and Torki, M. (2014). Al-bayan: An Arabic question answering system for the holy quran. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 57–64, Doha, Qatar, October. Association for Computational Linguistics. find
  4. Adouane, W. and Johansson, R. (2016). Gulf Arabic linguistic resource building for sentiment analysis. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2710–2715, Portorož, Slovenia, May. European Language Resources Association (ELRA). our card
  5. Alnajjar, K., Hämäläinen, M., Partanen, N., and Rueter, J. (2020). Automated prediction of medieval arabic diacritics. arXiv preprint arXiv:2010.05269. ↗ our card
  6. Alsaleh, A., Atwell, E., and Altahhan, A. (2021). Quranic verses semantic relatedness using AraBERT. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 185–190, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics. find
  7. Antoun, W., Baly, F., and Hajj, H. (2020). AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France, May. European Language Resource Association. ↗ our card
  8. Asai, A. and Choi, E. (2021). Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1492–1504, Online, August. Association for Computational Linguistics. ↗
  9. Belinkov, Y., Magidow, A., Romanov, M., Shmidman, A., and Koppel, M. (2016). Shamela: A large-scale historical Arabic corpus. In Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH), pages 45–53, Osaka, Japan, December. ↗
  10. The COLING 2016 Organizing Committee. find
  11. Ben Abdallah, N., Kchaou, S., and Bougares, F. (2020). Text and speech-based Tunisian Arabic sub-dialects identification. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6405–6411, Marseille, France, May. European Language Resources Association. our card
  12. Dukes, K. and Habash, N. (2010). Morphological annotation of quranic Arabic. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta, May. European Language Resources Association (ELRA). our card
  13. Habash, N., Eskander, R., and Hawwari, A. (2012). A morphological analyzer for Egyptian Arabic. In Proceedings of the Twelfth Meeting of the Special Interest Group on Computational Morphology and Phonology, pages 1–9, Montréal, Canada, June. Association for Computational Linguistics. our card
  14. Hämäläinen, M., Alnajjar, K., Partanen, N., and Rueter, J. (2021a). Never guess what I heard... rumor detection in Finnish news: a dataset and a baseline. In Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 39–44, Online, June. Association for Computational Linguistics. find
  15. Hämäläinen, M., Patpong, P., Alnajjar, K., Partanen, N., and Rueter, J. (2021b). Detecting depression in thai blog posts: a dataset and a baseline. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 20–25. find
  16. Hämäläinen, M. (2019). Uralicnlp: An nlp library for uralic languages. Journal of open source software. ↗
  17. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., and Habash, N. (2021). The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online), April. Association for Computational Linguistics. our card
  18. Jarrar, M., Habash, N., Akra, D., and Zalmout, N. (2014). Building a corpus for palestinian Arabic: a preliminary study. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 18–27, Doha, Qatar, October. Association for Computational Linguistics. find
  19. Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. In arXiv. ↗
  20. Loshchilov, I. and Hutter, F. (2017). Decoupled weight decay regularization. In arXiv. ↗
  21. Majadly, M. and Sagi, T. (2021). Dynamic ensembles in named entity recognition for historical Arabic texts. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 115–125, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics. find
  22. Mäkelä, E., Lagus, K., Lahti, L., Säily, T., Tolonen, M., Hämäläinen, M., Kaislaniemi, S., Nevalainen, T., et al. (2020). Wrangling with non-standard data. In Proceedings of the Digital Humanities in the Nordic Countries 5th Conference Riga, Latvia, October 21-23, 2020. CEUR-WS. org. find
  23. Malhas, R. and Elsayed, T. (2020). ¡i¿ayatec¡/i¿: Building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 19(6), oct. find
  24. Nishida, K., Nishida, K., Nagata, M., Otsuka, A., Saito, I., Asano, H., and Tomita, J. (2019). Answering while summarizing: Multi-task learning for multi-hop QA with evidence extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2335–2345, Florence, Italy, July. Association for Computational Linguistics. ↗
  25. Rücklé, A., Pfeiffer, J., and Gurevych, I. (2020). MultiCQA: Zero-shot transfer of self-supervised text matching models on a massive scale. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2471–2486, Online, November. Association for Computational Linguistics. find
  26. Safaya, A., Abdullatif, M., and Yuret, D. (2020). KUI-SAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online), December. International Committee for Computational Linguistics. find
  27. Sharaf, A.-B. and Atwell, E. (2012). QurAna: Corpus of the quran annotated with pronominal anaphora. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 130–137, Istanbul, Turkey, May. European Language Resources Association (ELRA). 7. find
  28. Language Resource References Artetxe, Mikel and Ruder, Sebastian and Yogatama, Dani. (2020). On the Cross-lingual Transferability of Monolingual Representations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. our card
  29. Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). our card
  30. Lewis, Patrick and Oguz, Barlas and Rinott, Ruty and Riedel, Sebastian and Schwenk, Holger. (2020). MLQA: Evaluating Cross-lingual Extractive Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. our card
  31. Malhas, Rana and Elsayed, Tamer. (2020). Ayatec: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP). find
  32. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November. Association for Computational Linguistics. ↗ our card
  33. Rajpurkar, Pranav and Jia, Robin and Liang, Percy. (2018). Know What You Don’t Know: Unanswerable Questions for SQuAD. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alnajjar-2022-harnessing-multilingual-resources-question-answering/