Elamur Research⌕
☰
home / papers / 2022 / Tafsir Dataset: A Novel Multi-Task Benchmark for Named Entity Recognition and Topic Modeling in Classical Arabic Literature

Tafsir Dataset: A Novel Multi-Task Benchmark for Named Entity Recognition and Topic Modeling in Classical Arabic Literature

Paper · 2022 · Sajawel Ahmed, Rob van der Goot, Misbahur Rehman et al.

Summary

Classical Arabic (CA) has a vast historical literature but almost no NLP resources. The authors address this with the Tafsir Dataset, built from Tafsir Al-Tabari, a large Quranic exegesis (Turki Edition, 2001). They converted the text to TEI and split it into sentences with a heuristic that uses editorial punctuation and then falls back on connective words. They annotated three independent layers: span-level named entities (PER, LOC, ORG, TME, OTH), sentence-level topics based on the classical fields of tafsir study, and span-level subtopics. The result has more than 51,000 sentences and is presented as the first large multi-task benchmark for CA. For evaluation, they trained a lightweight BiLSTM and used the MaChAmp multi-task toolkit with Arabic pretrained models. They also tested a script compression scheme that removes vowel marks and then letter dots, reducing the text to the 16-shape Rasm skeleton. NER reaches up to 95.58% F1, a new state of the art for CA-NER, while topic modeling proves difficult. The skeleton script performs close to the standard script, which suggests that much orthographic detail is not needed for these tasks.

Key points

  • The corpus has 51,704 sentences. NER counts are PER 176,105, LOC 5,583, ORG 22,026, TME 4,160 and OTH 12,453, far larger than ANERCorp (5,887 sentences) or AQMAR (2,646).
  • There are 15 topic classes, including fiqh, kalam, lugha and qiraat. The topic layer has 75,409 sentence-level annotations, and the subtopic layer has 27,037 span-level annotations.
  • The source edition fills 26 volumes and 18,594 pages. The sentence-splitting heuristic yields sentences of about 30 words on average.
  • Over 119 experimental setups were run with MaChAmp and models such as AraBERT, AraElectra and RemBERT. Adding a CRF layer helped, and the best CA-NER score was 95.58% F1.
  • Script compression shrinks the letter inventory from 280 unique characters in vocalized text to 28 in standard script and 16 in the skeleton form, while performance stays roughly stable. A web tool at linkedopentafsir.de offers more than 400 filters for non-technical scholars.

Interesting ideas

  • I1. The authors strip away layers of orthography one by one and retrain at each step, which shows how much surface detail a model really needs.
  • I2. When punctuation is missing, they split sentences at frequent clause-opening connectives such as "and", "so" and "then", producing units of usable length for downstream models.
  • I3. Their error analysis separates true model misses from gaps in the gold annotation, and most annotation gaps turn up among the false positives.
  • I4. Classical names appear first as long chains and later as short forms, and the model fails on very short sentences because it sees only one sentence while annotators saw the whole chapter.
  • I5. A learning curve over gold data in 5k-sentence steps showed whether more annotation still pays off, and the answer differed from earlier findings on other low-resource languages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Run the tagger and lemmatizer on progressively degraded English input (no case, no punctuation, collapsed inflections) to find which surface cues each decision actually depends on.
  • I2 (tokens): Add a fallback splitter for unpunctuated transcripts or run-on text that breaks at coordinating connectives, so the parser never receives an unmanageably long segment.
  • I3 (learn): Before counting a disagreement with silver data as a model error, check its false positives for missing gold labels and route those to repair rules rather than penalties.
  • I4 (world): Type entities against the document-level world, so a bare short name in a two-word sentence inherits the type and identity established at its earlier full mention.
  • I5 (learn): Plot such curves for each silver-data source before buying more LLM-teacher labels, and stop adding data for a task once its curve flattens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksNamed entity recognition · 200 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Topic modeling · 47 works

Cite

Cite as Sajawel Ahmed, Rob van der Goot, Misbahur Rehman, Carl Kruse, Ömer Özsoy, Alexander Mehler and Gemma Roig. 2022. Tafsir Dataset: A Novel Multi-Task Benchmark for Named Entity Recognition and Topic Modeling in Classical Arabic Literature. Proceedings of the 29th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ahmed2022tafsir,
  title = {{Tafsir Dataset: A Novel Multi-Task Benchmark for Named Entity Recognition and Topic Modeling in Classical Arabic Literature}},
  author = {Sajawel Ahmed, Rob {van der Goot}, Misbahur Rehman, Carl Kruse, Ömer Özsoy, Alexander Mehler and Gemma Roig},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  doi = {10.48448/58j4-et50},
  url = {https://doi.org/10.48448/58j4-et50}
}

References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sajawel Ahmed and Alexander Mehler. 2018. Resource-Size matters: Improving Neural Named Entity Recognition with Optimized Large Corpora. In Proceedings of the 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Florida, Orlando, USA. find
  2. Sajawel Ahmed, Misbahur Rehman, Joshua Tisch-lik, Carl Kruse, Edin Mahmutovic, and Ömer Özsoy. 2022. Linked Open Tafsir—Rekonstruktion der Entstehungsdynamik (en) des Korans mithilfe der Netzwerkmodellierung früher islamischer Über-lieferungen. In 8. Jahrestagung des Verbandes Digital Humanities im deutschsprachigen Raum (DHd). find
  3. Sajawel Ahmed, Manuel Stoeckel, Christine Driller, Adrian Pachzelt, and Alexander Mehler. 2019. BIOfid dataset: Publishing a German gold standard for named entity recognition in historical biodiversity literature. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 871–880, Hong Kong, China. Association for Computational Linguistics. find
  4. Jalal Al-Din Al-Suyuti. 1505. Al-itqan Fi ’ulum Al-Qur’an (The Perfect Guide to the Sciences of the Qu’ran). Garnet Publishing; Bilingual edition (May 1, 2012). find
  5. Amin Almuhanna and Jean-Francois Prunet. 2019. From Classical to Modern Arab Names and Back. Anthropological Linguistics, 61(4):405–458. Copyright - Copyright University of Nebraska Press Winter 2019; Last updated - 2021-10-04; Sub-jectsTermNotLitGenreText - Arabian Peninsula; Kuwait. ↗
  6. Yassine Benajiba, Paolo Rosso, and José Miguel Benedíruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 143–153. Springer. find
  7. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. ↗
  8. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022. ↗
  9. Armin Burkhardt. 2004. 2004. Nomen est omen? : zur Semantik der Eigennamen. In Landesheimat-bund Sachsen-Anhalt e. V. (Hrsg.): "Magdeburger Namenlandschaft" : Orts- und Personennamen der Stadt und Region Magdeburg. find
  10. B Barla Cambazoglu, Mark Sanderson, Falk Scholer, and Bruce Croft. 2021. A review of public datasets in question answering research. In ACM SIGIR Forum, volume 54, pages 1–23. ↗
  11. ACM New York, NY, USA. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  12. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  13. Daria Dzendzik, Jennifer Foster, and Carl Vogel. 2021. English machine reading comprehension datasets: A survey. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8784–8804, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  14. Anoual El Kah and Imad Zeroual. 2021. Arabic topic identification: A decade scoping review. In E3S Web of Conferences, volume 297, page 01058. EDP Sciences. ↗
  15. Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012. Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. In LREC.
  16. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural Architectures for Named Entity Recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260–270. Association for Computational Linguistics. ↗
  17. Wang Ling, Chris Dyer, Alan Black, and Isabel Trancoso. 2015. Two/Too Simple Adaptations of word2vec for Syntax Problems. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.
  18. Hajer Maraoui, Kais Haddar, and Laurent Romary. 2017. Encoding prototype of al-hadith al-shareef in tei. In International Conference on Arabic Language Processing, pages 217–229. Springer. find
  19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  20. Matthew Thomas Miller, Maxim G. Romanov, and Sarah Bowen Savant. 2018. Digitizing the textual heritage of the premodern islamicate world: Principles and plans. International Journal of Middle East Studies, 50(1):103–109. ↗
  21. Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A Smith. 2012. Recall-oriented learning of named entities in arabic wikipedia. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 162–173.
  22. Lawrence Philips. 1990. Hanging on the metaphone. In Computer Language, volume 7, pages 39–43. find
  23. Barbara Plank, Kristian Nørgaard Jensen, and Rob van der Goot. 2020. DaN+: Danish nested named entities and lexical normalization. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6649–6662, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  24. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789, Melbourne, Australia. Association for Computational Linguistics. ↗
  25. Stefan Schweter and Sajawel Ahmed. 2019. Deep-EOS: General-Purpose Neural Networks for Sentence Boundary Detection. In Proceedings of the 15th Conference on Natural Language Processing (KONVENS). find
  26. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. ↗
  27. Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multi-task learning in NLP. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 176–197, Online. Association for Computational Linguistics. ↗
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ahmed-2022-tafsir-dataset-novel-multi-task/