Elamur Research⌕
☰
home / papers / 2022 / Challenging the Transformer-based models with a Classical Arabic dataset: Quran and Hadith

Challenging the Transformer-based models with a Classical Arabic dataset: Quran and Hadith

Summary

Transformer models score near or above human level on popular similarity benchmarks. The authors argue these scores overstate real understanding for two reasons. Random train/test splits leak instances between sets, and benchmark sentences are easy to read. Both problems are worse for Classical Arabic, which has few datasets and hard morphology, orthography and rhetoric. The authors frame the task as binary classification: do two Classical Arabic passages share the same underlying message? They built a new test set of Quran-verse/Hadith-teaching pairs. Each link comes from reputable Islamic scholars, and non-related pairs were added as negatives. Models are fine-tuned on an extended version of the Qursim dataset, which holds only Quran–Quran pairs, and then evaluated on the stylistically different Quran–Hadith pairs. Several monolingual, bilingual and multilingual state-of-the-art models were compared. The authors conclude that these models still fall short at capturing deep meaning in complex, low-resource texts. The datasets are released publicly.

Key points

  • Only three Arabic semantic similarity/relatedness datasets existed: Qursim (CA, 7,679 pairs, 2012), Q2Q from NSURL (MSA, 15,712 pairs, 2019) and the SemEval-2017 STS Arabic set (MSA, 1,104 pairs). Qursim was the only Classical Arabic one.
  • The Quran–Hadith (QH) test set was built in five stages:
    • choose expert sources that state relatedness explicitly;
    • extract the pairs;
    • check each text against the original corpora so that full verses and teachings are kept;
    • generate non-related pairs;
    • strip punctuation and diacritics.
  • The QH pairs come from two sources:
    • Sahih Albukhari, where section headings that cite a Quran verse are paired with the Hadith Matn (the teaching without its chain of narrators) of that section, taken from the LK-Hadith-Corpus;
    • the archived Fatwas of Abdul-Aziz ibn Baz.
  • Qursim was itself mined from Ibn Kathir's commentary, which explains each verse by pointing to related verses. The extended Qursim was used for fine-tuning and validation, and QH was reserved for testing.
  • The authors cite earlier work reporting that random splits can produce up to 70% instance overlap between training and test data. This is part of why training and testing here use different text types.

Interesting ideas

  • I1. Train on pairs from one text type (Quran–Quran) and test on a related but stylistically distinct type (Quran–Hadith), so that scores reflect generalization rather than memorized overlap. — For Mova (learn): Silver-trained components could be scored on a deliberately different genre, for example training on expository prose and testing on fables, as a built-in check against leakage.
  • I2. Instead of crowdsourcing labels that need expertise, harvest them from the structure of expert-authored documents, such as section headings citing a verse or rulings that cite both texts. — For Mova (answer): Textbook chapters whose headings and worked exercises point to specific passages could yield question–evidence pairs for school-course tests without manual annotation.
  • I3. Check every extracted fragment against the source corpus so that each item is a complete unit rather than a truncated excerpt. — For Mova (learn): When assembling silver sentences or frame examples from scraped material, re-anchoring each item to its original document would catch clipped sentences before they reach training.
  • I4. Treat multi-topic sources with caution, because a link between two passages may be unclear once they are lifted out of their wider context. — For Mova (answer): QA items for fables and tales could be filtered so the answer is supported by the given passage alone, not by context lost during extraction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages

Institutions

🎓University of LeedsLeeds · United Kingdom19 works · 1 290 citations🎓King Saud UniversityRiyadh · Saudi Arabia13 works · 290 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{altammami2022challenging,
  title = {{Challenging the Transformer-based models with a Classical Arabic dataset: Quran and Hadith}},
  author = {Altammami, Shatha and Atwell, Eric},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/2j3tytcry54x},
  url = {https://doi.org/10.63317/2j3tytcry54x}
}

References

40 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abbas, N. H. (2009). Quran ’search for a concept’ tool and website. Master’s thesis, University of Leeds. find
  2. Abdul-Raof, H. (2013). Qur’an translation: Discourse, texture and exegesis. Routledge. find
  3. Alsaleh, A. N., Atwell, E., and Altahhan, A. (2021). Quranic verses semantic relatedness using arabert. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 185–190. Leeds. find
  4. Alshammeri, M., Atwell, E., and Alsalka, M. A. (2020). Quranic topic modelling using paragraph vectors. In Proceedings of SAI Intelligent Systems Conference, pages 218–230. Springer. find
  5. Altammami, S., Atwell, E., and Alsalka, A. (2020a). Constructing a bilingual hadith corpus using a segmentation tool. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3390–3398. find
  6. Altammami, S., Atwell, E., and Alsalka, A. (2020b). Towards a joint ontology of quran and hadith. International Journal on Islamic Applications in Computer Science And Technology. find
  7. Alyafeai, Z., Masoud, M., Ghaleb, M., and Al-shaibani, M. S. (2021). Masader: Metadata sourcing for arabic text and speech data resources. find our card
  8. Antoun, W., Baly, F., and Hajj, H. (2020). Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗ our card
  9. Bashir, M. H., M Azmi, A., Nawaz, H., Zaghouani, W., Diab, M., Al-Fuqaha, A., and Qadir, J. (2021). Arabic natural language processing for qur’anic research: A systematic review. find
  10. Belinkov, Y., Magidow, A., Barrón-Cedeño, A., Shmidman, A., and Romanov, M. (2019). Studying the history of the arabic language: language technology and a large-scale historical corpus. Language Resources and Evaluation, 53(4):771–805. find
  11. Bender, E. M. and Koller, A. (2020). Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198. find our card
  12. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find our card
  13. Bounhas, I. (2019). On the usage of a classical arabic corpus as a language resource: related research and key challenges. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(3):1–45. find
  14. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. (2017). Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055. ↗ our card
  15. Chandrasekaran, D. and Mago, V. (2020). Domain specific complex sentence (dscs) semantic similarity dataset. arXiv preprint arXiv:2010.12637. ↗
  16. Chicco, D. and Jurman, G. (2020). The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC genomics, 21(1):1–13. find
  17. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  18. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗ our card
  19. Elangovan, A., He, J., and Verspoor, K. (2021). Memorization vs. generalization: quantifying data leakage in nlp performance evaluation. arXiv preprint arXiv:2102.01818. ↗
  20. Habash, N. Y. (2010). Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. find
  21. Hakkoum, A. and Raghay, S. (2016). Ontological approach for semantic modeling and querying. International Journal on Islamic Applications in Computer Science and Technology, pages 37–37. find
  22. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., and Habash, N. (2021). The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678. ↗ our card
  23. Kidwai, A.-R. (1987). Translating the untranslatable: a survey of english translations of the quran. The Muslim World Book Review, 7(4):66–71. find
  24. Lan, W., Chen, Y., Xu, W., and Ritter, A. (2020). An empirical study of pre-trained transformers for arabic information extraction. arXiv preprint arXiv:2004.14519. ↗
  25. Le, Q. and Mikolov, T. (2014). Distributed representations of sentences and documents. In International conference on machine learning, pages 1188–1196. PMLR. find our card
  26. Lewis, P., Stenetorp, P., and Riedel, S. (2020). Question and answer test-train overlap in open-domain question answering datasets. arXiv preprint arXiv:2008.02637. ↗
  27. Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., Zamparelli, R., et al. (2014). A sick cure for the evaluation of compositional distributional semantic models. In Lrec, pages 216–223. Reykjavik. find our card
  28. Mozannar, H., Maamary, E., El Hajal, K., and Hajj, H. (2019). Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy, aug. Association for Computational Linguistics. find
  29. Mubarak, H., Darwish, K., Magdy, W., Elsayed, T., and Al-Khalifa, H. (2020). Overview of OSACT4 Arabic offensive language detection shared task. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 48– 52, Marseille, France, may. European Language Resource Association. find
  30. Nagoudi, E. M., Ferrero, J., and Schwab, D. (2017). Lim-lig at semeval-2017 task1: Enhancing the semantic similarity for arabic sentences with vectors weighting. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 134–138. find
  31. Obeid, O., Zalmout, N., Khalifa, S., Taji, D., Oudah, M., Alhafni, B., Inoue, G., Eryani, F., Erdmann, A., and Habash, N. (2020). CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022– 7032, Marseille, France, May. European Language Resources Association. find our card
  32. Peuriekeu, Y. M., Noyum, V. D., Feudjio, C., Goktug, A., and Fokoue, E. (2021). A text mining discovery of similarities and dissimilarities among sacred scriptures. arXiv preprint arXiv:2102.04421. ↗
  33. Qahl, S. H. M. (2014). An automatic similarity detection engine between sacred texts using text mining and similarity measures. Rochester Institute of Technology. find
  34. Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. ↗ our card
  35. Saeed, S., Haider, S., and Rajput, Q. (2020). On finding similar verses from the holy quran using word embeddings. In 2020 International Conference on Emerging Trends in Smart Technologies (ICETST), pages 1–6. IEEE. find
  36. Safaya, A., Abdullatif, M., and Yuret, D. (2020). Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059. find our card
  37. Seelawi, H., Mustafa, A., Al-Bataineh, H., Farhan, W., and Al-Natsheh, H. T. (2019). Nsurl-2019 task 8: Semantic question similarity in arabic. In Proceedings of The First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019-Short Papers, pages 1–8. find
  38. Sharaf, A.-B. and Atwell, E. (2012). Qursim: A corpus for evaluation of relatedness in short texts. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2295–2302. find
  39. Varghese, N. and Punithavalli, M. (2019). Lexical and semantic analysis of sacred texts using machine learning and natural language processing. International Journal of Scientific & Technology Research, 8(12):3133–3140. find
  40. Verma, M. (2017). Lexical analysis of religious texts using text mining and machine learning tools. International Journal of Computer Applications, 168(8):39–45. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/altammami-2022-challenging-transformer-based-models-classical/