Elamur Research⌕
☰
home / papers / 2024 / AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis

AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis

Summary

Arabic poetry is difficult for NLP. Verses follow strict prosodic meters and their variants, a whole poem keeps one rhyme, and the language differs from everyday prose. Earlier computational work mostly addressed only meter detection and sentiment, using rule systems or recurrent networks. Faisal Qarah introduces AraPoemBERT, a BERT-base model pretrained from scratch exclusively on poetry. It is trained on AraPoems, a newly compiled corpus of over 2.09 million verses gathered online and tagged with meter, sub-meter, poet, rhyme and topic. After fine-tuning, the model is compared with five general-purpose Arabic models on five classification tasks. Three of these tasks are new: poet gender, sub-meter and rhyme. AraPoemBERT is best or close to best on most tasks, with 99.03% on meter and 78.95% on sentiment, both above earlier studies. The study suggests that a narrow, genre-specific pretraining corpus can beat broader models in a specialised domain. The released model, data and code also give future poetry tasks a baseline to measure against.

Key points

  • The comparison models are AraBERTv0.1-base, AraBERTv1-base, QARiB, ARBERT and CAMeLBERT-CA. AraPoemBERT scores 97.79% on sub-meter, 99.34% on gender and 97.73% on rhyme. CAMeLBERT-CA is slightly ahead on rhyme with 97.76%.
  • Sub-meters combine a classical meter with a variant such as Majzuu, Mashture or Mukhala, giving 25 classes in the report. The rare classes do poorly, for example Baseet mukhala with an F1 of 0.0785 and Wafer majzuu with 0.1257.
  • For gender, the poets were labelled manually from their names: 5,383 poets, of whom 5,023 are male and 360 female. The overall accuracy is high, but the Female class reaches only F1 0.2246 (recall 0.1318).
  • Rhyme uses 31 labels: the 28 Arabic letters plus Laa, Taa Marbutah and Waw Hamza. About 24.5% of Laa samples are predicted as Lam.
  • The resources are public: the model at huggingface faisalq/bert-base-arapoembert, the AraPoems dataset on Dataverse (doi 10.7910/DVN/PJPWOY), and the code on GitHub.

Interesting ideas

  • I1. Pretraining a model from scratch only on the target genre can beat larger general models, because the vocabulary and patterns fit the domain. — For Mova (tree): An encoder for gparse could be pretrained on fables, tales and school-course text, then compared with a general encoder on LAS for these genres.
  • I2. Joining a coarse label with its variant into one fine label (meter plus variant becomes sub-meter) lets a single model be scored at two levels of granularity. — For Mova (frames): Frame predictions could be scored on both the VerbNet class and its subclass, so we see whether errors are coarse or only fine-grained.
  • I3. The per-class reports show that a headline accuracy near 99% can hide a minority class that almost always fails. — For Mova (learn): Evaluation reports should always include per-class and macro F1 next to accuracy, so rare dependency relations or roles cannot hide behind frequent ones.
  • I4. Annotating at a higher unit (the poet) and passing the label down to millions of verses creates a new task cheaply. — For Mova (learn): Document-level or source-level facts (genre, author, register) could be attached once and copied to every sentence, giving silver labels for genre-aware training.
  • I5. Checking the confusion matrix for classes that differ only by spelling (Laa versus Lam) shows where the labels themselves are almost the same. — For Mova (tokens): We could look for confusions between orthographic variants and contractions in lemmatization, and either merge such classes or add explicit rules for them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.12392 · arXiv 2403.12392

@misc{qarah2024arapoembert,
  title = {{AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis}},
  author = {Qarah, Faisal},
  year = {2024},
  doi = {10.48550/arxiv.2403.12392},
  eprint = {2403.12392},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.12392}
}

References

38 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zwettler, M.: Oral Tradition of Classical Arabic Poetry: Its Character and Implications. The Ohio State University Press, Columbus, Ohio (1978) find
  2. Arberry, A.J.: Arabic Poetry: a Primer for Students. CUP Archive, Cambridge (1965) find
  3. Scott, H.: Pegs, cords, and ghuls: Meter of classical arabic poetry. PhD thesis (2010) find
  4. Atiq, A.: Elm al-arud wal qafiah (in arabic). Dar Alnahda, Beirut, Lebanon (1987) find
  5. Ahmed Abdel Aziz, A.M.D., Juma Al Kubaisi, M.S.: Types of causes in the ancient arabic prosodic lesson.(in arabic). Journal of Anbar University for Languages & Literature/Magallat Gami’at Al-Anbar Li-Lugat Wa-al-Adabl (26) (2018) find
  6. Sowayan, S.A., Al-Shaibani, M., Al-Zumar, S.: Nabati Poetry The Taste of the People and The Authority of The Text (in Arabic). Abu Dhabi Authority for Culture and Heritage, Abu Dhabi (2010) find
  7. Alabbas, M., Khalaf, Z.A., Khashan, K.M.: Basrah: an automatic system to identify the meter of arabic poetry. Natural Language Engineering 20(1), 131–149 (2014) find
  8. Abuata, B., Al-Omari, A.: A rule-based algorithm for the detection of arud meter in classical arabic poetry. International Arab Journal of Information Technology 15(4), 1–5 (2018) find
  9. Alnagdawi, M.A., Rashideh, H., Aburumman, F.: Finding arabic poem meter using context free grammar. Journal of Communication and Computer Engineering 3(1), 52–59 (2013) find
  10. Berkani, A., Holzer, A., Stoffel, K.: Pattern matching in meter detection of arabic classical poetry. In: 2020 IEEE/ACS 17th International Conference on Computer Systems and Applications (AICCSA), pp. 1–8 (2020). IEEE find
  11. Yousef, W.A., Ibrahime, O.M., Madbouly, T.M., Mahmoud, M.A.: Learning meters of arabic and english poems with recurrent neural networks: a step forward for language understanding and synthesis. arXiv preprint arXiv:1905.05700 (2019) ↗
  12. Al-Shaibani, M.S., Alyafeai, Z., Ahmad, I.: Meter classification of arabic poems using deep bidirectional recurrent neural networks. Pattern Recognition Letters 136, 1–7 (2020) find
  13. Abandah, G.A., Khedher, M.Z., Abdel-Majeed, M.R., Mansour, H.M., Hulliel, S.F., Bisharat, L.M.: Classifying and diacritizing arabic poems using deep recurrent neural networks. Journal of King Saud University-Computer and Information Sciences 34(6), 3775–3788 (2022) find
  14. Abboushi, O., Azzeh, M.: Toward fluent arabic poem generation based on fine-tuning aragpt2 transformer. Arabian Journal for Science and Engineering, 1–13 (2023) find
  15. Antoun, W., Baly, F., Hajj, H.: Aragpt2: Pre-trained transformer for arabic language generation. arXiv preprint arXiv:2012.15520 (2020) ↗
  16. Mohammad, I.: Naive bayes for classical arabic poetry classification. Al-Nahrain Journal of Science 12(4), 217–225 (2009) find
  17. Alsharif, O., Alshamaa, D., Ghneim, N.: Emotion classification in arabic poetry using machine learning. International Journal of Computer Applications 65(16) (2013) find
  18. Ahmed, M.A., Hasan, R.A., Ali, A.H., Mohammed, M.A.: The classification of the modern arabic poetry using machine learning. TELKOMNIKA (Telecommunication Computing Electronics and Control) 17(5), 2667–2674 (2019) find
  19. Shahriar, S., Al Roken, N., Zualkernan, I.: Classification of arabic poetry emotions using deep learning. Computers 12(5), 89 (2023) find
  20. Antoun, W., Baly, F., Hajj, H.: Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104 (2020) ↗ our card
  21. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017) find
  22. El-Khair, I.A.: 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033 (2016) ↗
  23. Zeroual, I., Goldhahn, D., Eckart, T., Lakhouaja, A.: Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In: Proceedings of the Fourth Arabic Natural Language Processing Workshop, pp. 175–182 (2019) find
  24. Abdelali, A., Darwish, K., Durrani, N., Mubarak, H.: Farasa: A fast and furious segmenter for arabic. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pp. 11–16 (2016) find our card
  25. Chowdhury, S.A., Abdelali, A., Darwish, K., Soon-Gyo, J., Salminen, J., Jansen, B.J.: Improving arabic text categorization using transformer training diversification. In: Proceedings of the Fifth Arabic Natural Language Processing Workshop, pp. 226–236 (2020) find
  26. Abdul-Mageed, M., Elmadany, A., Nagoudi, E.M.B.: Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785 (2020) ↗ our card
  27. Suárez, P.J.O., Sagot, B., Romary, L.: Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In: 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7) (2019). Leibniz-Institut für Deutsche Sprache find
  28. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., Habash, N.: The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678 (2021) ↗ our card
  29. Alammary, A.S.: Bert models for arabic text classification: a systematic review. Applied Sciences 12(11), 5720 (2022) find
  30. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) ↗
  31. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016) ↗
  32. Almausua. https://poetry.dctabudhabi.ae/. Accessed: 2023-01-10 ↗
  33. Aldiwan. https://www.aldiwan.net/. Accessed: 2023-01-19 ↗
  34. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020) find our card
  35. Zerrouki, T.: Pyarabic, an Arabic language library for Python. Pyarabic (2010) find
  36. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) ↗
  37. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017) ↗
  38. Zhang, Y., Warstadt, A., Li, H.-S., Bowman, S.R.: When do you need billions of words of pretraining data? arXiv preprint arXiv:2011.04946 (2020) ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qarah-2024-arapoembert-pretrained-language-model-arabic/