Elamur Research⌕
☰
home / papers / 2023 / Noor-Ghateh: A Benchmark Dataset for Evaluating Arabic Word Segmenters in Hadith Domain

Noor-Ghateh: A Benchmark Dataset for Evaluating Arabic Word Segmenters in Hadith Domain

Summary

The paper tackles the lack of morphologically annotated resources for classical religious Arabic, specifically Hadith texts. Arabic words often bundle several pieces (prefixes, a stem and suffixes) into a single orthographic token, so segmentation is a first step for most downstream analysis. The authors built Noor-Ghateh, a corpus of roughly 223,690 words drawn from the Hadith collection "Shariat al-Islam", with segmentation labels assigned by human experts. To show how the resource behaves as a benchmark, they ran three existing segmenters (Farasa, CAMeL and ALP) on it and on two smaller reference sets: the NAFIS stemming gold standard and a portion of the Quranic Arabic Corpus. A word counts as correct only if every predicted segment matches the expert's split. All three tools scored about 0.8 on Noor-Ghateh, and the scores were very close to each other. Their ranking also changed from one dataset to another. The authors argue that Noor-Ghateh is larger and more lexically varied than earlier Hadith-related resources. They present it as the reference benchmark for Hadith segmentation, though only a sample is promised for public release.

Key points

  • Noor-Ghateh contains about 223,690 expert-annotated words from the book "Shariat al-Islam" (labelled "Sharaye" in the results table).
  • The comparison benchmarks were NAFIS, a gold set of 172 words for Arabic stemmer evaluation, and 7,043 words of the Quranic Arabic Corpus covering the Hamd and Baqarah Surahs.
  • Accuracy on NAFIS / Quran / Sharaye:
    • Farasa: 0.59 / 0.76 / 0.81
    • CAMeL: 0.68 / 0.80 / 0.81
    • ALP: 0.65 / 0.81 / 0.79
  • The scoring is strict: a word's segmentation is correct only when all of its components agree with the human annotation.
  • Future work is a Seq-to-Seq segmenter with attention, to be evaluated against this benchmark. Only a sample of the dataset is planned for open release.

Interesting ideas

  • I1. Scoring a word as correct only when the whole split is right, rather than giving partial credit per segment, makes the metric reflect what a downstream consumer actually receives. — For Mova (tokens): Score English clitic and contraction splitting ("can't", "John's", "gonna") per word with all-pieces-must-match, so a half-correct split shows up as an error before it reaches the tree.
  • I2. Running the same tools on a tiny, a mid-sized and a large in-domain benchmark showed that rankings flip between datasets, so one test set can mislead. — For Mova (learn): Report each parser or tagger change on several held-out sets of different genres (school texts, fables, news) and treat a win on only one set as unconfirmed.
  • I3. Building a benchmark from one long, coherent text in a narrow register, annotated by domain experts, captures vocabulary and constructions that general corpora under-represent. — For Mova (tree): A small expert-checked treebank of fables and tales could serve as an in-domain gold set to measure LAS drift of the perceptron parser and gparse away from news-heavy UD data.
  • I4. Validating a new gold set by running several independent off-the-shelf systems on it shows whether the data is usable and how hard it is, before any new model exists. — For Mova (learn): When adding a new silver or gold corpus, first run the existing witness parsers on it to set a baseline and spot annotation-convention mismatches.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

1 paper of our library cites it (of the 140884 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.09630 · arXiv 2307.09630

@misc{alshuhayeb2023noor,
  title = {{Noor-Ghateh: A Benchmark Dataset for Evaluating Arabic Word Segmenters in Hadith Domain}},
  author = {AlShuhayeb, Huda and Minaei-Bidgoli, Behrouz and Shenassa, Mohammad E. and Hossayni, Sayyed-Ali},
  year = {2023},
  doi = {10.48550/arxiv.2307.09630},
  eprint = {2307.09630},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.09630}
}

References

58 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tim Buckwalter. Buckwalter arabic morphological analyzer version 1.0. Linguistic Data Consortium, University of Pennsylvania, pages 86–93, 2002. find
  2. T Buckwalter. Buckwalter arabic morphological analyzer (bama) version 2.0. linguistic data consortium (ldc) catalogue number ldc2004l02. Technical report, ISBN1-58563-324-0, 2004. find
  3. D Graff, M Maamouri, B Bouziri, S Krouna, S Kulick, and T Buckwalter. Standard arabic morphological analyzer (sama). Linguistic Data Consortium LDC2009E73, 2010. find
  4. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. The penn arabic treebank: Building a large-scale annotated arabic corpus. In NEMLAR conference on Arabic language resources and tools, volume 27, pages 466–467. Cairo, 2004. find
  5. Khaled Elghamry. A constraint-based algorithm for the identification of arabic roots. In Proceedings of the 1st Midwest Computational Linguistics Colloquium. Indiana Univ. Bloomington, 2004. find
  6. Nizar Y Habash and Owen C Rambow. Magead: A morphological analyzer and generator for the arabic dialects. 2006. find our card
  7. Paul Rodrigues and Damir Cavar. Learning arabic morphology using statistical constraint-satisfaction models. AMSTERDAM STUDIES IN THE THEORY AND HISTORY OF LINGUISTIC SCIENCE SERIES 4, 289:63, 2007. find
  8. Nizar Habash. Arabic morphological representations for machine translation. In Arabic computational morphology: Knowledge-based and empirical methods, pages 263–285. Springer, 2007. find
  9. Otakar Smrz. Elixirfm–implementation of functional arabic morphology. In Proceedings of the 2007 workshop on computational approaches to Semitic languages: common issues and resources, pages 1–8, 2007. find our card
  10. Ezra Daya, Dan Roth, and Shuly Wintner. Identifying semitic roots: Machine learning with linguistic constraints. Computational Linguistics, 34(3):429–448, 2008. find our card
  11. Dan Roth. Learning to resolve natural language ambiguities: A unified approach. Aaai/iaai, 1998:806–813, 1998. find
  12. Benjamin Snyder and Regina Barzilay. Unsupervised multilingual learning for morphological segmentation. In Proceedings of acl-08: hlt, pages 737–745, 2008. find
  13. Hoifung Poon, Colin Cherry, and Kristina Toutanova. Unsupervised morphological segmentation with log-linear models. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 209–217, 2009. find
  14. Abderrahim Boudlal, Mohamed Ould Abdallahi Ould Bebah, Abdelhak Lakhouaja, Azzeddine Mazroui, and Abdelouafi Meziane. A markovian approach for arabic root extraction. Int. Arab J. Inf. Technol., 8(1):91–98, 2011. find
  15. Mohammed Attia, Pavel Pecina, Antonio Toral, Lamia Tounsi, and Josef van Genabith. An open-source finite state morphological transducer for modern standard arabic. In Proceedings of the 9th International Workshop on Finite State Methods and Natural Language Processing, pages 125–133, 2011. find our card
  16. Michelle Fullwood and Timothy O’Donnell. Learning non-concatenative morphology. In Proceedings of the Fourth Annual Workshop on Cognitive Modeling and Computational Linguistics (CMCL), pages 21–27, 2013. find
  17. Bilal Khaliq and John A Carroll. Induction of root and pattern lexicon for unsupervised morphological analysis of arabic. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1012–1016, 2013. find
  18. Salam Khalifa, Sara Hassan, and Nizar Habash. A morphological analyzer for gulf arabic verbs. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 35–45, 2017. find
  19. Salam Khalifa, Nasser Zalmout, and Nizar Habash. Morphological analysis and disambiguation for gulf arabic: The interplay between resources and methods. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3895–3904, 2020. find our card
  20. Dima Taji, Salam Khalifa, Ossama Obeid, Fadhl Eryani, and Nizar Habash. An arabic morphological analyzer and generator with copious features. In Proceedings of the fifteenth workshop on computational research in phonetics, phonology, and morphology, pages 140–150, 2018. find our card
  21. Salam Khalifa, Nizar Habash, Dana Abdulrahim, and Sara Hassan. A large scale corpus of gulf arabic. pages 4282–4289, 2016. find
  22. Nizar Habash, Ramy Eskander, and Abdelati Hawwari. A morphological analyzer for egyptian arabic. In Proceedings of the twelfth meeting of the special interest group on computational morphology and phonology, pages 1–9, 2012. find our card
  23. Mourad Gridach and Noureddine Chenfour. Developing a new system for arabic morphological analysis and generation. In Proceedings of the 2nd Workshop on South Southeast Asian Natural Language Processing (WSSANLP), pages 52–57, 2011. find
  24. Nasser Zalmout and Nizar Habash. Don’t throw those morphological analyzers away just yet: Neural morphological disambiguation for arabic. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 704–713, 2017. find our card
  25. Abderrahim Boudlal, Abdelhak Lakhouaja, Azzeddine Mazroui, Abdelouafi Meziane, MOAO Bebah, and Mostafa Shoul. Alkhalil morpho sys1: A morphosyntactic analysis system for arabic texts. In International Arab conference on information technology, pages 1–6. Elsevier Science Inc New York, NY, 2010. find
  26. Mohamed Boudchiche, Azzeddine Mazroui, Mohamed Ould Abdallahi Ould Bebah, Abdelhak Lakhouaja, and Abderrahim Boudlal. Alkhalil morpho sys 2: A robust arabic morpho-syntactic analyzer. Journal of King Saud University-Computer and Information Sciences, 29(2):141–146, 2017. find
  27. Lamia Hadrich Belguith Inès Zribi, Mariem Ellouze Khemakhem. Morphological analysis of tunisian dialect. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, 2013. find
  28. Sarah Alkuhlani and Nizar Habash. A corpus for modeling morpho-syntactic agreement in arabic: Gender, number and rationality. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 357–362, 2011. find our card
  29. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC, volume 14, pages 1094–1101, 2014. find our card
  30. Nizar Habash and Owen Rambow. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of the 43rd annual meeting of the association for computational linguistics (ACL’05), pages 573–580, 2005. find our card
  31. Nizar Habash, Owen Rambow, and Ryan Roth. Mada+ tokan: A toolkit for arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. In Proceedings of the 2nd international conference on Arabic language resources and tools (MEDAR), Cairo, Egypt, volume 41, page 62, 2009. find
  32. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. Automatic processing of modern standard arabic text. In Arabic computational morphology: Knowledge-based and Empirical Methods, pages 159–179. Springer, 2007. find
  33. Sameh Alansary. Basma: Bibalex standard arabic morphological analyzer. The Egyptian Journal of Language Engineering, 3(1):24–33, 2016. find
  34. Salam Khalifa, Nasser Zalmout, and Nizar Habash. Yamama: Yet another multi-dialect arabic morphological analyzer. In Proceedings of COLING 2016, the 26th international conference on computational linguistics: system demonstrations, pages 223–227, 2016. find
  35. H Kilany, H Gadalla, H Arram, A Yacoub, A El-Habashi, and C McLemore. Egyptian colloquial arabic lexicon. LDC catalog number LDC99L22, 2002. find
  36. Faisal Alshargi, Shahd Dibas, Sakhar Alkhereyf, Reem Faraj, Basmah Abdulkareem, Sane Yagi, Ouafaa Kacha, Nizar Habash, and Owen Rambow. Morphologically annotated corpora for seven arabic dialects: Taizi, sanaani, najdi, jordanian, syrian, iraqi and moroccan. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 137–147, 2019. find our card
  37. Rabiaa MOULAY OMAR, Aissa ELDJOUZI, Mohamed Amine CHERAGUI, et al. Development of a segmenter for Algerian dialect texts. PhD thesis, University Ahmed Draya-Adrar, 2023. find
  38. Majdi Sawalha, Eric Atwell, and Mohammad AM Abushariah. Salma: standard arabic language morphological analysis. In 2013 1st International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pages 1–6. IEEE, 2013. find
  39. Hanen Himdi and Fatmah Assiri. Tasaheel: An arabic automative textual analysis tool-all in one. IEEE Access, 2023. find
  40. Waleed Nazih, Amany Fashwan, Amr El-Gendy, and Yasser Hifny. Ibn-ginni: An improved morphological analyzer for arabic. ACM Transactions on Asian and Low-Resource Language Information Processing, 23(2):1–22, 2024. find
  41. Faisal Qarah and Tawfeeq Alsanoosy. A comprehensive analysis of various tokenizers for arabic large language models. Applied Sciences, 14(13):5696, 2024. find
  42. Jan Hajic, Otakar Smrz, Petr Zemánek, Jan Šnaidauf, and Emanuel Beška. Prague arabic dependency treebank: Development in data and tools. In Proc. of the NEMLAR Intern. Conf. on Arabic Language Resources and Tools, volume 1, 2004. find
  43. Petr Zemánek. Clara (corpus linguae arabicae): An overview. In Proceedings of ACL/EACL Workshop on Arabic Language, 2001. find
  44. Imad Zeroual and Abdelhak Lakhouaja. A new quranic corpus rich in morphosyntactical information. International Journal of Speech Technology, 19:339–346, 2016. find
  45. Kais Dukes and Nizar Habash. Morphological annotation of quranic arabic. In LREC, pages 2530–2536, 2010. find our card
  46. Judith Dror, Dudu Shaharabani, Rafi Talmon, and Shuly Wintner. Morphological analysis of the qur’an. Literary and linguistic Computing, 19(4):431–452, 2004. find
  47. Eric Atwell. Corpus resources for learning arabic to understand the quran. In Higher Education Academy workshop on “The Role of Corpora in LSP ( Language for Specific Purposes) Learning and Teaching”, 2012. find
  48. Imad Zeroual and Abdelhak Lakhouaja. Clitiques-stemmer: nouveau stemmer pour la langue arabe. In The First National Doctoral Symposium On Arabic Language Engineering (JDILA’2014), 2014. find
  49. Mohamed Osman Hegazi, Anwer Hilal, and Mohammad Alhawarat. Fine-grained quran dataset. International Journal of Advanced Computer Science and Applications (IJACSA), 6(12):262–267, 2015. find
  50. Nizar Habash, Behrang Mohit, Ossama Obeid, Kemal Oflazer, Nadi Tomeh, and Wajdi Zaghouani. Qalb: Qatar arabic language bank. In Proceedings of Qatar Annual Research Conference (ARC-2013), pages ICTP–032, Doha, Qatar, 2013. find
  51. Taha Zerrouki and Amar Balla. Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems. Data in Brief, 11:147–151, 2017. find
  52. Robert Parker et al. Arabic Gigaword Fifth Edition LDC2011T11. Philadelphia, 2011. find
  53. Yousef Altaher, Ali Fadel, Mazen Alotaibi, Alyazidi, et al. Masader plus: A new interface for exploring+ 500 arabic nlp datasets. arXiv preprint arXiv:2208.00932, 2022. ↗ our card
  54. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16, 2016. find our card
  55. Will Monroe, Spence Green, and Christopher D Manning. Word segmentation of informal arabic with domain adaptation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 206–211, 2014. find
  56. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. Camel tools: An open source python toolkit for arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, 2020. find our card
  57. Abed Alhakim Freihat, Gabor Bella, Hamdy Mubarak, and Fausto Giunchiglia. A single-model approach for arabic segmentation, pos tagging, and named entity recognition. In 2018 2nd International Conference on Natural Language and Speech Processing (ICNLSP), pages 1–8. IEEE, 2018. find
  58. Driss Namly, R. Tajmout, Karim Bouzoubaa, and Lahsen Abouenour. Nafis: A gold standard corpus for arabic stemmers evaluation. 11 2016. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alshuhayeb-2023-noor-ghateh-benchmark-dataset-evaluating/